From: Martin Jansa <martin.jansa@gmail.com>
To: openembedded-core@lists.openembedded.org
Cc: tomasz.dziendzielski@gmail.com, Martin Jansa <Martin.Jansa@gmail.com>
Subject: [PATCH] externalsrc.bbclass: fix git repo detection
Date: Fri, 21 Oct 2022 01:03:51 +0200 [thread overview]
Message-ID: <20221020230351.2712106-1-Martin.Jansa@gmail.com> (raw)
* fix issue introduced in:
https://git.openembedded.org/openembedded-core/commit/?id=95fbac8dcad6c93f4c9737e9fe13e92ab6befa09
* it added check for s_dir + git-dir (typically '.git') isn't
the same as ${TOPDIR} + git-dir, but due to copy-paste issue
it was just comparing it with s_dir + git-dir again, resulting
in most external repos (where git-dir is '.git') to be processed
as regular directory (not taking advantage of git write-tree).
* normally this wouldn't be an issue, but for big repo with a lot of
files this added a lot of checksums in:
d.setVarFlag('do_compile', 'file-checksums', '${@srctree_hash_files(d)}')
and I mean *a lot, e.g. in chromium build it was 380227 paths
which still wouldn't that bad, but the checksum processing in
siggen.py isn't trivial and just looping through all these
checksums takes very long time (over 1000sec on fast NVME drive
with warm cache) and then
https://git.openembedded.org/bitbake/commit/?id=b4975d2ecf615ac4c240808fbc5a3f879a93846b
made the processing a bit more complicated and the loop in
get_taskhash() function took 6448sec and to make things worse
there was no output from bitbake during that time, so even with -DDD
it looks like this:
DEBUG: virtual/libgles2 resolved to: mesa (langdale/oe-core/meta/recipes-graphics/mesa/mesa_22.2.0.bb)
Bitbake still alive (no events for 600s). Active tasks:
Bitbake still alive (no events for 1200s). Active tasks:
Bitbake still alive (no events for 1800s). Active tasks:
Bitbake still alive (no events for 2400s). Active tasks:
Bitbake still alive (no events for 3000s). Active tasks:
Bitbake still alive (no events for 3600s). Active tasks:
Bitbake still alive (no events for 4200s). Active tasks:
Bitbake still alive (no events for 4800s). Active tasks:
Bitbake still alive (no events for 5400s). Active tasks:
Bitbake still alive (no events for 6000s). Active tasks:
DEBUG: Starting bitbake-worker
without -DDD it will get stuck for almost 2 hours in:
"Initialising tasks..."
before it finally writes sstate summary like:
"Sstate summary: Wanted 3102 Local 0 Mirrors 0 Missed 3102 Current 1483 (0% match, 32% complete)"
* fix the copy&paste typo to use git work-tree in most cases, but
be aware that this issue still exists for huge local source
trees not in git
[YOCTO #14942]
Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
---
meta/classes-recipe/externalsrc.bbclass | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/meta/classes-recipe/externalsrc.bbclass b/meta/classes-recipe/externalsrc.bbclass
index 06a9548a20..5905342589 100644
--- a/meta/classes-recipe/externalsrc.bbclass
+++ b/meta/classes-recipe/externalsrc.bbclass
@@ -212,8 +212,8 @@ def srctree_hash_files(d, srcdir=None):
try:
git_dir = os.path.join(s_dir,
subprocess.check_output(['git', '-C', s_dir, 'rev-parse', '--git-dir'], stderr=subprocess.DEVNULL).decode("utf-8").rstrip())
- top_git_dir = os.path.join(s_dir, subprocess.check_output(['git', '-C', d.getVar("TOPDIR"), 'rev-parse', '--git-dir'],
- stderr=subprocess.DEVNULL).decode("utf-8").rstrip())
+ top_git_dir = os.path.join(d.getVar("TOPDIR"),
+ subprocess.check_output(['git', '-C', d.getVar("TOPDIR"), 'rev-parse', '--git-dir'], stderr=subprocess.DEVNULL).decode("utf-8").rstrip())
if git_dir == top_git_dir:
git_dir = None
except subprocess.CalledProcessError:
--
2.38.1
reply other threads:[~2022-10-20 23:04 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20221020230351.2712106-1-Martin.Jansa@gmail.com \
--to=martin.jansa@gmail.com \
--cc=openembedded-core@lists.openembedded.org \
--cc=tomasz.dziendzielski@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox