From: ecordonnier@snap.com
To: openembedded-core@lists.openembedded.org
Cc: Etienne Cordonnier <ecordonnier@snap.com>
Subject: [PATCH v3 1/3] archiver.bbclass: archive shallow git sources which have no clone
Date: Tue, 17 Nov 2026 13:24:55 +0100 [thread overview]
Message-ID: <20261117122457.435864-2-ecordonnier@snap.com> (raw)
In-Reply-To: <20261117122457.435864-1-ecordonnier@snap.com>
From: Etienne Cordonnier <ecordonnier@snap.com>
In the original mode, do_ar_original archives a source if the fetcher's
localpath is a file, or a directory (which is unpacked and tarred). For git,
localpath was the clone in DL_DIR/git2, which a shallow fetch (BB_GIT_SHALLOW)
does not create when it uses the gitshallow tarball: since bitbake commit
457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is
what a shallow fetch does by default, and it also happens when the shallow
tarball is fetched from a mirror. The source was then silently skipped and not
archived at all.
The git fetcher now returns the shallow tarball as localpath in this case.
Copying it would make the archive differ depending on how the source was
fetched, so unpack git sources like a clone, which uses the shallow tarball.
AI-Generated: Claude Sonnet 5.5
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
meta/classes/archiver.bbclass | 9 ++++-
meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++
2 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 0400d6e373..d1a8dadec7 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -204,10 +204,15 @@ python do_ar_original() {
fetch = bb.fetch.Fetch(urls, ld)
tarball_suffix = {}
for url in fetch.urls:
+ ud = fetch.ud[url]
+ is_git = isinstance(ud.method, bb.fetch.git.Git)
local = fetch.localpath(url).rstrip("/");
- if os.path.isfile(local):
+ # The localpath of a git source is the shallow tarball, instead of the
+ # clone, when the fetch used the shallow tarball only. Unpack it like
+ # a clone instead of archiving the tarball itself.
+ if os.path.isfile(local) and not is_git:
shutil.copy(local, ar_outdir)
- elif os.path.isdir(local):
+ elif os.path.isdir(local) or (is_git and os.path.isfile(local)):
tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
fetch.unpack(tmpdir, (url,))
# To handle recipes with more than one source, we add the "name"
diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
index 8c5aa75e9b..84f13839c9 100644
--- a/meta/lib/oeqa/selftest/cases/archiver.py
+++ b/meta/lib/oeqa/selftest/cases/archiver.py
@@ -7,6 +7,8 @@
import os
import glob
import re
+import shutil
+import tarfile
from oeqa.utils.commands import bitbake, get_bb_vars
from oeqa.selftest.case import OESelftestTestCase
@@ -176,6 +178,52 @@ class Archiver(OESelftestTestCase):
self._test_archiver_mode('original', 'ed-1.21.1.tar.lz')
+ def test_archiver_mode_original_git_shallow(self):
+ """
+ Test that the archiver archives a shallow git source with
+ `ARCHIVER_MODE[src] = "original"` when only the shallow tarball is
+ available, but not the clone. This is what happens with the fast
+ initial shallow fetch, or when the shallow tarball is fetched from a
+ mirror.
+ """
+ target = 'gitrepotest'
+
+ features = 'INHERIT += "archiver"\n'
+ features += 'ARCHIVER_MODE[src] = "original"\n'
+ features += 'BB_GIT_SHALLOW = "1"\n'
+ features += 'BB_GENERATE_SHALLOW_TARBALLS = "1"\n'
+ features += 'DL_DIR = "${TOPDIR}/downloads-shallow"\n'
+ self.write_config(features)
+
+ # Fetch to create the shallow tarball, then remove the clone if the
+ # fetcher fell back to a full clone, so that the second fetch has to
+ # use the shallow tarball. cleansstate makes sure that the archive is
+ # created again and not restored from sstate.
+ bitbake('-c cleansstate %s' % target)
+ bitbake('-c fetch %s' % target)
+ bb_vars = get_bb_vars(['DL_DIR'])
+ self.track_for_cleanup(bb_vars['DL_DIR'])
+ self.assertTrue(glob.glob(os.path.join(bb_vars['DL_DIR'], 'gitshallow_*.tar.gz')),
+ 'Missing shallow tarball in DL_DIR')
+ shutil.rmtree(os.path.join(bb_vars['DL_DIR'], 'git2'))
+ bitbake('-c cleansstate %s' % target)
+
+ bitbake('-c deploy_archives %s' % target)
+
+ names = self._get_original_git_archive_names(target)
+ self.assertTrue([n for n in names if n.endswith('/COPYING')], 'Archive has no source files: %s' % names[:20])
+
+ def _get_original_git_archive_names(self, target):
+ """
+ Return the names of the files in the original source archive of the
+ git source of target.
+ """
+ bb_vars = get_bb_vars(['DEPLOY_DIR_SRC'], target)
+ archives = glob.glob(os.path.join(bb_vars['DEPLOY_DIR_SRC'], '*', '%s-*' % target, '%s-*.tar.*' % target))
+ self.assertEqual(len(archives), 1, 'Expected one source archive for %s, found %s' % (target, archives))
+ with tarfile.open(archives[0]) as tar:
+ return tar.getnames()
+
def test_archiver_mode_patched(self):
"""
Test that the archiver works with `ARCHIVER_MODE[src] = "patched"`.
--
2.43.0
next prev parent reply other threads:[~2026-11-17 12:25 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-17 12:24 [PATCH v3 0/3] archiver.bbclass: archive shallow git sources which have no clone ecordonnier
2026-11-17 12:24 ` ecordonnier [this message]
2026-11-17 12:24 ` [PATCH v3 2/3] archiver.bbclass: remove .git from original git source archives ecordonnier
2026-11-17 12:24 ` [PATCH v3 3/3] archiver.bbclass: fail on sources which cannot be archived ecordonnier
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20261117122457.435864-2-ecordonnier@snap.com \
--to=ecordonnier@snap.com \
--cc=openembedded-core@lists.openembedded.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox