* [bitbake][kirkstone][2.0][PATCH 1/8] runqueue: Ensure deferred tasks are sorted by multiconfig
2022-10-04 15:54 [bitbake][kirkstone][2.0][PATCH 0/8] Patch review Steve Sakoman
@ 2022-10-04 15:54 ` Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 2/8] runqueue: Improve deadlock warning messages Steve Sakoman
` (6 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Steve Sakoman @ 2022-10-04 15:54 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
We have to prefer one multiconfig over another when deferring tasks, else
we'll have cross-linked build trees and nothing will be able to build.
In the original population code, we sort like this but we don't after
rehashing. Ensure we have the same sorting after rehashing toa void
deadlocks.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 27228c7f026acb8ae9e1211d0486ffb7338123a2)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/runqueue.py | 13 ++++++++-----
1 file changed, 8 insertions(+), 5 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 111dc0ee..a23f7a89 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2523,11 +2523,14 @@ class RunQueueExecute:
if update_tasks:
self.sqdone = False
- for tid in [t[0] for t in update_tasks]:
- h = pending_hash_index(tid, self.rqdata)
- if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]:
- self.sq_deferred[tid] = self.sqdata.hashes[h]
- bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h]))
+ for mc in sorted(self.sqdata.multiconfigs):
+ for tid in sorted([t[0] for t in update_tasks]):
+ if mc_from_tid(tid) != mc:
+ continue
+ h = pending_hash_index(tid, self.rqdata)
+ if h in self.sqdata.hashes and tid != self.sqdata.hashes[h]:
+ self.sq_deferred[tid] = self.sqdata.hashes[h]
+ bb.note("Deferring %s after %s" % (tid, self.sqdata.hashes[h]))
update_scenequeue_data([t[0] for t in update_tasks], self.sqdata, self.rqdata, self.rq, self.cooker, self.stampcache, self, summary=False)
for (tid, harddepfail, origvalid) in update_tasks:
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [bitbake][kirkstone][2.0][PATCH 2/8] runqueue: Improve deadlock warning messages
2022-10-04 15:54 [bitbake][kirkstone][2.0][PATCH 0/8] Patch review Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 1/8] runqueue: Ensure deferred tasks are sorted by multiconfig Steve Sakoman
@ 2022-10-04 15:54 ` Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 3/8] runqueue: Drop deadlock breaking force fail Steve Sakoman
` (5 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Steve Sakoman @ 2022-10-04 15:54 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Tweak the deadlock breaking messages to be explict about which task is
blocked on which other task. The messages currently imply it is "freeing"
the blocking task which is confusing.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit cf7f60b83adaded180f6717cb4681edc1d65b66d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/runqueue.py | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index a23f7a89..0d89740b 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2254,10 +2254,12 @@ class RunQueueExecute:
# No more tasks can be run. If we have deferred setscene tasks we should run them.
if self.sq_deferred:
- tid = self.sq_deferred.pop(list(self.sq_deferred.keys())[0])
- logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s" % tid)
- if tid not in self.runq_complete:
- self.sq_task_failoutright(tid)
+ deferred_tid = list(self.sq_deferred.keys())[0]
+ blocking_tid = self.sq_deferred.pop(deferred_tid)
+ logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s blocked by %s" % (deferred_tid, blocking_tid))
+ if blocking_tid not in self.runq_complete:
+ logger.warning("Failing blocking task %s" % (blocking_tid))
+ self.sq_task_failoutright(blocking_tid)
return True
if self.failed_tids:
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [bitbake][kirkstone][2.0][PATCH 3/8] runqueue: Drop deadlock breaking force fail
2022-10-04 15:54 [bitbake][kirkstone][2.0][PATCH 0/8] Patch review Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 1/8] runqueue: Ensure deferred tasks are sorted by multiconfig Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 2/8] runqueue: Improve deadlock warning messages Steve Sakoman
@ 2022-10-04 15:54 ` Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 4/8] siggen: Fix insufficent entropy in sigtask file names Steve Sakoman
` (4 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Steve Sakoman @ 2022-10-04 15:54 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
I'm 99% certain this failing of a scenequeue task corrupts runqueue and
causes all kinds of breakage. I'd rather runqueue deadlocked than corrupted
and did weird things so drop this code.
We've seen builds where the deadlock triggers and it then tries to run tasks
where the SQ task already ran with very confusing failures. It is likely it
is this code causing it.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 8efced47fcb47851a370fd6786df6fb377f99963)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/runqueue.py | 3 ---
1 file changed, 3 deletions(-)
diff --git a/lib/bb/runqueue.py b/lib/bb/runqueue.py
index 0d89740b..48e25401 100644
--- a/lib/bb/runqueue.py
+++ b/lib/bb/runqueue.py
@@ -2257,9 +2257,6 @@ class RunQueueExecute:
deferred_tid = list(self.sq_deferred.keys())[0]
blocking_tid = self.sq_deferred.pop(deferred_tid)
logger.warning("Runqeueue deadlocked on deferred tasks, forcing task %s blocked by %s" % (deferred_tid, blocking_tid))
- if blocking_tid not in self.runq_complete:
- logger.warning("Failing blocking task %s" % (blocking_tid))
- self.sq_task_failoutright(blocking_tid)
return True
if self.failed_tids:
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [bitbake][kirkstone][2.0][PATCH 4/8] siggen: Fix insufficent entropy in sigtask file names
2022-10-04 15:54 [bitbake][kirkstone][2.0][PATCH 0/8] Patch review Steve Sakoman
` (2 preceding siblings ...)
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 3/8] runqueue: Drop deadlock breaking force fail Steve Sakoman
@ 2022-10-04 15:54 ` Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 5/8] bitbake: Add copyright headers where missing Steve Sakoman
` (3 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Steve Sakoman @ 2022-10-04 15:54 UTC (permalink / raw)
To: bitbake-devel
From: Joshua Watt <JPEWhacker@gmail.com>
Signature generation uses mkstemp() to get a file descriptor to a unique
file and then write the signature into it. However, the unique file name
generation in glibc is based on the system timestamp, which means that
with highly parallel builds it is more likely than one might expect
expected that a conflict will occur between two different builder nodes.
When operating over NFS (such as a shared sstate cache), this can cause
race conditions and rare failures (particularly with NFS servers that
may not correctly implement O_EXCL).
The signature generation code is particularly susceptible to races since
a single "sigtask." prefix used for all signatures from all tasks, which
makes collision even more likely.
To work around this, add an internal implementation of mkstemp() that
adds additional truly random entropy to the file name to eliminate
conflicts.
Signed-off-by: Joshua Watt <JPEWhacker@gmail.com>
Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com>
(cherry picked from commit 97955f3c1c738aa4b4478a6ec10a08094ffc689d)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/siggen.py | 2 +-
lib/bb/utils.py | 21 +++++++++++++++++++++
2 files changed, 22 insertions(+), 1 deletion(-)
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index 9fa568f6..bd6fc204 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -419,7 +419,7 @@ class SignatureGeneratorBasic(SignatureGenerator):
bb.error("Taskhash mismatch %s versus %s for %s" % (computed_taskhash, self.taskhash[tid], tid))
sigfile = sigfile.replace(self.taskhash[tid], computed_taskhash)
- fd, tmpfile = tempfile.mkstemp(dir=os.path.dirname(sigfile), prefix="sigtask.")
+ fd, tmpfile = bb.utils.mkstemp(dir=os.path.dirname(sigfile), prefix="sigtask.")
try:
with bb.compress.zstd.open(fd, "wt", encoding="utf-8", num_threads=1) as f:
json.dump(data, f, sort_keys=True, separators=(",", ":"), cls=SetEncoder)
diff --git a/lib/bb/utils.py b/lib/bb/utils.py
index 95b3c898..92d44c52 100644
--- a/lib/bb/utils.py
+++ b/lib/bb/utils.py
@@ -28,6 +28,8 @@ import signal
import collections
import copy
import ctypes
+import random
+import tempfile
from subprocess import getstatusoutput
from contextlib import contextmanager
from ctypes import cdll
@@ -1756,3 +1758,22 @@ def is_local_uid(uid=''):
if str(uid) == line_split[2]:
return True
return False
+
+def mkstemp(suffix=None, prefix=None, dir=None, text=False):
+ """
+ Generates a unique filename, independent of time.
+
+ mkstemp() in glibc (at least) generates unique file names based on the
+ current system time. When combined with highly parallel builds, and
+ operating over NFS (e.g. shared sstate/downloads) this can result in
+ conflicts and race conditions.
+
+ This function adds additional entropy to the file name so that a collision
+ is independent of time and thus extremely unlikely.
+ """
+ entropy = "".join(random.choices("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890", k=20))
+ if prefix:
+ prefix = prefix + entropy
+ else:
+ prefix = tempfile.gettempprefix() + entropy
+ return tempfile.mkstemp(suffix=suffix, prefix=prefix, dir=dir, text=text)
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [bitbake][kirkstone][2.0][PATCH 5/8] bitbake: Add copyright headers where missing
2022-10-04 15:54 [bitbake][kirkstone][2.0][PATCH 0/8] Patch review Steve Sakoman
` (3 preceding siblings ...)
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 4/8] siggen: Fix insufficent entropy in sigtask file names Steve Sakoman
@ 2022-10-04 15:54 ` Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 6/8] Fix npm to use https rather than http Steve Sakoman
` (2 subsequent siblings)
7 siblings, 0 replies; 9+ messages in thread
From: Steve Sakoman @ 2022-10-04 15:54 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
Where copyright headers were not present, add them to make things
clear.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 1aa338a216350a2751fff52f866039343e9ac013)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
bin/bitbake-prserv | 2 ++
bin/bitbake-worker | 2 ++
bin/git-make-shallow | 2 ++
lib/bb/COW.py | 2 ++
lib/bb/asyncrpc/__init__.py | 2 ++
lib/bb/asyncrpc/client.py | 2 ++
lib/bb/asyncrpc/serv.py | 2 ++
lib/bb/codeparser.py | 2 ++
lib/bb/compress/_pipecompress.py | 2 ++
lib/bb/compress/lz4.py | 2 ++
lib/bb/compress/zstd.py | 2 ++
lib/bb/daemonize.py | 2 ++
lib/bb/exceptions.py | 2 ++
lib/bb/fetch2/osc.py | 2 ++
lib/bb/process.py | 2 ++
lib/bb/siggen.py | 2 ++
lib/bb/tests/compression.py | 2 ++
lib/bb/tests/cooker.py | 2 ++
lib/bblayers/__init__.py | 2 ++
lib/bblayers/action.py | 2 ++
lib/bblayers/common.py | 2 ++
lib/bblayers/layerindex.py | 2 ++
lib/bblayers/query.py | 2 ++
lib/prserv/__init__.py | 2 ++
lib/prserv/client.py | 2 ++
lib/prserv/db.py | 2 ++
lib/prserv/serv.py | 2 ++
lib/toaster/manage.py | 2 ++
28 files changed, 56 insertions(+)
diff --git a/bin/bitbake-prserv b/bin/bitbake-prserv
index 323df66d..5be42f3c 100755
--- a/bin/bitbake-prserv
+++ b/bin/bitbake-prserv
@@ -1,5 +1,7 @@
#!/usr/bin/env python3
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/bin/bitbake-worker b/bin/bitbake-worker
index 9d850ec7..2f3e9f72 100755
--- a/bin/bitbake-worker
+++ b/bin/bitbake-worker
@@ -1,5 +1,7 @@
#!/usr/bin/env python3
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/bin/git-make-shallow b/bin/git-make-shallow
index 1d00fbf1..d0532c5a 100755
--- a/bin/git-make-shallow
+++ b/bin/git-make-shallow
@@ -1,5 +1,7 @@
#!/usr/bin/env python3
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/COW.py b/lib/bb/COW.py
index 23c22b65..76bc08a3 100644
--- a/lib/bb/COW.py
+++ b/lib/bb/COW.py
@@ -3,6 +3,8 @@
#
# Copyright (C) 2006 Tim Ansell
#
+# SPDX-License-Identifier: GPL-2.0-only
+#
# Please Note:
# Be careful when using mutable types (ie Dict and Lists) - operations involving these are SLOW.
# Assign a file to __warn__ to get warnings about slow operations.
diff --git a/lib/bb/asyncrpc/__init__.py b/lib/bb/asyncrpc/__init__.py
index c2f2b3c0..9a85e996 100644
--- a/lib/bb/asyncrpc/__init__.py
+++ b/lib/bb/asyncrpc/__init__.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/asyncrpc/client.py b/lib/bb/asyncrpc/client.py
index 34960197..881434d2 100644
--- a/lib/bb/asyncrpc/client.py
+++ b/lib/bb/asyncrpc/client.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/asyncrpc/serv.py b/lib/bb/asyncrpc/serv.py
index b4cffff2..e14df18e 100644
--- a/lib/bb/asyncrpc/serv.py
+++ b/lib/bb/asyncrpc/serv.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/codeparser.py b/lib/bb/codeparser.py
index 3b3c3b41..9d66d3ae 100644
--- a/lib/bb/codeparser.py
+++ b/lib/bb/codeparser.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/compress/_pipecompress.py b/lib/bb/compress/_pipecompress.py
index 5de17a82..4a403d62 100644
--- a/lib/bb/compress/_pipecompress.py
+++ b/lib/bb/compress/_pipecompress.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
# Helper library to implement streaming compression and decompression using an
diff --git a/lib/bb/compress/lz4.py b/lib/bb/compress/lz4.py
index 0f6bc51a..88b09893 100644
--- a/lib/bb/compress/lz4.py
+++ b/lib/bb/compress/lz4.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/compress/zstd.py b/lib/bb/compress/zstd.py
index 50c42133..cdbbe9d6 100644
--- a/lib/bb/compress/zstd.py
+++ b/lib/bb/compress/zstd.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/daemonize.py b/lib/bb/daemonize.py
index 4957bfd4..76894044 100644
--- a/lib/bb/daemonize.py
+++ b/lib/bb/daemonize.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/exceptions.py b/lib/bb/exceptions.py
index ecbad599..801db9c8 100644
--- a/lib/bb/exceptions.py
+++ b/lib/bb/exceptions.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/fetch2/osc.py b/lib/bb/fetch2/osc.py
index eb0f82c8..bf4c2f05 100644
--- a/lib/bb/fetch2/osc.py
+++ b/lib/bb/fetch2/osc.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
"""
diff --git a/lib/bb/process.py b/lib/bb/process.py
index be2c15a1..4c7b6d39 100644
--- a/lib/bb/process.py
+++ b/lib/bb/process.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/siggen.py b/lib/bb/siggen.py
index bd6fc204..9a20fc8e 100644
--- a/lib/bb/siggen.py
+++ b/lib/bb/siggen.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/tests/compression.py b/lib/bb/tests/compression.py
index d3ddf67f..95af3f96 100644
--- a/lib/bb/tests/compression.py
+++ b/lib/bb/tests/compression.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bb/tests/cooker.py b/lib/bb/tests/cooker.py
index c82d4b7b..9e524ae3 100644
--- a/lib/bb/tests/cooker.py
+++ b/lib/bb/tests/cooker.py
@@ -1,6 +1,8 @@
#
# BitBake Tests for cooker.py
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bblayers/__init__.py b/lib/bblayers/__init__.py
index 4e7c09da..78efd297 100644
--- a/lib/bblayers/__init__.py
+++ b/lib/bblayers/__init__.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bblayers/action.py b/lib/bblayers/action.py
index 6723e2c6..454c2514 100644
--- a/lib/bblayers/action.py
+++ b/lib/bblayers/action.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bblayers/common.py b/lib/bblayers/common.py
index 6c76ef35..f7b9cee3 100644
--- a/lib/bblayers/common.py
+++ b/lib/bblayers/common.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bblayers/layerindex.py b/lib/bblayers/layerindex.py
index 79365162..0ac8fd2e 100644
--- a/lib/bblayers/layerindex.py
+++ b/lib/bblayers/layerindex.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/bblayers/query.py b/lib/bblayers/query.py
index 525d4f0d..9142ec44 100644
--- a/lib/bblayers/query.py
+++ b/lib/bblayers/query.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/prserv/__init__.py b/lib/prserv/__init__.py
index 9961040b..38ced818 100644
--- a/lib/prserv/__init__.py
+++ b/lib/prserv/__init__.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/prserv/client.py b/lib/prserv/client.py
index a3f19dda..69ab7a4a 100644
--- a/lib/prserv/client.py
+++ b/lib/prserv/client.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/prserv/db.py b/lib/prserv/db.py
index 2710d4a2..b4bda707 100644
--- a/lib/prserv/db.py
+++ b/lib/prserv/db.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/prserv/serv.py b/lib/prserv/serv.py
index 0a20b927..c686b206 100644
--- a/lib/prserv/serv.py
+++ b/lib/prserv/serv.py
@@ -1,4 +1,6 @@
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
diff --git a/lib/toaster/manage.py b/lib/toaster/manage.py
index ae32619d..f8de49c2 100755
--- a/lib/toaster/manage.py
+++ b/lib/toaster/manage.py
@@ -1,5 +1,7 @@
#!/usr/bin/env python3
#
+# Copyright BitBake Contributors
+#
# SPDX-License-Identifier: GPL-2.0-only
#
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [bitbake][kirkstone][2.0][PATCH 6/8] Fix npm to use https rather than http
2022-10-04 15:54 [bitbake][kirkstone][2.0][PATCH 0/8] Patch review Steve Sakoman
` (4 preceding siblings ...)
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 5/8] bitbake: Add copyright headers where missing Steve Sakoman
@ 2022-10-04 15:54 ` Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 7/8] gitsm: Error out if submodule refers to parent repo Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 8/8] asyncrpc/client: Fix unix domain socket chdir race issues Steve Sakoman
7 siblings, 0 replies; 9+ messages in thread
From: Steve Sakoman @ 2022-10-04 15:54 UTC (permalink / raw)
To: bitbake-devel
From: Neil Horman <nhorman@gmail.com>
Hit this error while building nlf-native recently:
{
"error": {
"summary": "URI malformed",
"detail": ""
}
}
Some poking about led me to discover that:
1) The npm.py tool replaces npm:// with http://, not https://
2) Some versions of the npm tool don't handle 301 redirects properly,
choosing to display the above error instead when using the default
nodejs registry
It would be good to go fix npm to handle the redirect properly, but it
seems like it would also be good to assume secure http when contacting a
registry, hence, this patch
Signed-off-by: Neil Horman <nhorman@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 2cd76e8aabe4e803c760e60f06cfe1f470714ec7)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/fetch2/npm.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/lib/bb/fetch2/npm.py b/lib/bb/fetch2/npm.py
index 8f7c10ac..8a179a33 100644
--- a/lib/bb/fetch2/npm.py
+++ b/lib/bb/fetch2/npm.py
@@ -156,7 +156,7 @@ class Npm(FetchMethod):
raise ParameterError("Invalid 'version' parameter", ud.url)
# Extract the 'registry' part of the url
- ud.registry = re.sub(r"^npm://", "http://", ud.url.split(";")[0])
+ ud.registry = re.sub(r"^npm://", "https://", ud.url.split(";")[0])
# Using the 'downloadfilename' parameter as local filename
# or the npm package name.
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [bitbake][kirkstone][2.0][PATCH 7/8] gitsm: Error out if submodule refers to parent repo
2022-10-04 15:54 [bitbake][kirkstone][2.0][PATCH 0/8] Patch review Steve Sakoman
` (5 preceding siblings ...)
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 6/8] Fix npm to use https rather than http Steve Sakoman
@ 2022-10-04 15:54 ` Steve Sakoman
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 8/8] asyncrpc/client: Fix unix domain socket chdir race issues Steve Sakoman
7 siblings, 0 replies; 9+ messages in thread
From: Steve Sakoman @ 2022-10-04 15:54 UTC (permalink / raw)
To: bitbake-devel
From: Pavel Zhukov <pavel@zhukoff.net>
If submodule refers to specific revision of the parent repository it
causes deadlock in bitbake locking mechanism (lock is acquired to fetch
the parent and cannot be released before all submodules are fetched).
raise FetchError in such situation to prevent deadlocking.
[Yocto 14045]
Signed-off-by: Pavel Zhukov <pavel@zhukoff.net>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 0361ecf7eb82c386a9842cf1f3cb706c0a112e77)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/fetch2/gitsm.py | 3 +++
1 file changed, 3 insertions(+)
diff --git a/lib/bb/fetch2/gitsm.py b/lib/bb/fetch2/gitsm.py
index c1950e48..25d5db0e 100644
--- a/lib/bb/fetch2/gitsm.py
+++ b/lib/bb/fetch2/gitsm.py
@@ -115,6 +115,9 @@ class GitSM(Git):
# This has to be a file reference
proto = "file"
url = "gitsm://" + uris[module]
+ if "{}{}".format(ud.host, ud.path) in url:
+ raise bb.fetch2.FetchError("Submodule refers to the parent repository. This will cause deadlock situation in current version of Bitbake." \
+ "Consider using git fetcher instead.")
url += ';protocol=%s' % proto
url += ";name=%s" % module
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread* [bitbake][kirkstone][2.0][PATCH 8/8] asyncrpc/client: Fix unix domain socket chdir race issues
2022-10-04 15:54 [bitbake][kirkstone][2.0][PATCH 0/8] Patch review Steve Sakoman
` (6 preceding siblings ...)
2022-10-04 15:54 ` [bitbake][kirkstone][2.0][PATCH 7/8] gitsm: Error out if submodule refers to parent repo Steve Sakoman
@ 2022-10-04 15:54 ` Steve Sakoman
7 siblings, 0 replies; 9+ messages in thread
From: Steve Sakoman @ 2022-10-04 15:54 UTC (permalink / raw)
To: bitbake-devel
From: Richard Purdie <richard.purdie@linuxfoundation.org>
The connect_unix() call had a bug where if a relative path to a socket
was passed (which the non-async client always does), and the current
working directory was changed after the initial call, it would fail to
reconnect if it became disconnected, since the socket couldn't be found
relative to the new current working directory.
To work around this, change the socket connection for UNIX domain
sockets to be synchronous and change current working before connecting.
This isn't ideal since the connection could block the entire event loop,
but in practice this shouldn't happen since the socket are local files
anyway.
Help debugging and resolving from Joshua Watt.
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 5964bb67bb20df7f411ee0650cf189504a05cf25)
Signed-off-by: Steve Sakoman <steve@sakoman.com>
---
lib/bb/asyncrpc/client.py | 22 +++++++++++++---------
1 file changed, 13 insertions(+), 9 deletions(-)
diff --git a/lib/bb/asyncrpc/client.py b/lib/bb/asyncrpc/client.py
index 881434d2..fa042bbe 100644
--- a/lib/bb/asyncrpc/client.py
+++ b/lib/bb/asyncrpc/client.py
@@ -31,7 +31,17 @@ class AsyncClient(object):
async def connect_unix(self, path):
async def connect_sock():
- return await asyncio.open_unix_connection(path)
+ # AF_UNIX has path length issues so chdir here to workaround
+ cwd = os.getcwd()
+ try:
+ os.chdir(os.path.dirname(path))
+ # The socket must be opened synchronously so that CWD doesn't get
+ # changed out from underneath us so we pass as a sock into asyncio
+ sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM, 0)
+ sock.connect(os.path.basename(path))
+ finally:
+ os.chdir(cwd)
+ return await asyncio.open_unix_connection(sock=sock)
self._connect_sock = connect_sock
@@ -150,14 +160,8 @@ class Client(object):
setattr(self, m, self._get_downcall_wrapper(downcall))
def connect_unix(self, path):
- # AF_UNIX has path length issues so chdir here to workaround
- cwd = os.getcwd()
- try:
- os.chdir(os.path.dirname(path))
- self.loop.run_until_complete(self.client.connect_unix(os.path.basename(path)))
- self.loop.run_until_complete(self.client.connect())
- finally:
- os.chdir(cwd)
+ self.loop.run_until_complete(self.client.connect_unix(path))
+ self.loop.run_until_complete(self.client.connect())
@property
def max_chunk(self):
--
2.25.1
^ permalink raw reply related [flat|nested] 9+ messages in thread