Openembedded Core Discussions
 help / color / mirror / Atom feed
* [PATCH v3 0/3] archiver.bbclass: archive shallow git sources which have no clone
@ 2026-11-17 12:24 ecordonnier
  2026-11-17 12:24 ` [PATCH v3 1/3] " ecordonnier
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: ecordonnier @ 2026-11-17 12:24 UTC (permalink / raw)
  To: openembedded-core; +Cc: Etienne Cordonnier

From: Etienne Cordonnier <ecordonnier@snap.com>

As feedback to the v2 of this patch, it was requested to modify the fetcher to ensure that
localpath() exists when do_fetch completes, in order to avoid the need
for special handling in the archiver bbclass.

This v3 series addresses this feedback, and requires the bitbake
change "fetch/git: return the shallow tarball as localpath when there is no clone"

Etienne Cordonnier (3):
  archiver.bbclass: archive shallow git sources which have no clone
  archiver.bbclass: remove .git from original git source archives
  archiver.bbclass: fail on sources which cannot be archived

 meta/classes/archiver.bbclass            | 23 +++++-
 meta/lib/oeqa/selftest/cases/archiver.py | 91 ++++++++++++++++++++++++
 2 files changed, 112 insertions(+), 2 deletions(-)

-- 
2.43.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH v3 1/3] archiver.bbclass: archive shallow git sources which have no clone
  2026-11-17 12:24 [PATCH v3 0/3] archiver.bbclass: archive shallow git sources which have no clone ecordonnier
@ 2026-11-17 12:24 ` ecordonnier
  2026-11-17 12:24 ` [PATCH v3 2/3] archiver.bbclass: remove .git from original git source archives ecordonnier
  2026-11-17 12:24 ` [PATCH v3 3/3] archiver.bbclass: fail on sources which cannot be archived ecordonnier
  2 siblings, 0 replies; 4+ messages in thread
From: ecordonnier @ 2026-11-17 12:24 UTC (permalink / raw)
  To: openembedded-core; +Cc: Etienne Cordonnier

From: Etienne Cordonnier <ecordonnier@snap.com>

In the original mode, do_ar_original archives a source if the fetcher's
localpath is a file, or a directory (which is unpacked and tarred). For git,
localpath was the clone in DL_DIR/git2, which a shallow fetch (BB_GIT_SHALLOW)
does not create when it uses the gitshallow tarball: since bitbake commit
457288b2f ("fetch2/git: Add support for fast initial shallow fetch") this is
what a shallow fetch does by default, and it also happens when the shallow
tarball is fetched from a mirror. The source was then silently skipped and not
archived at all.

The git fetcher now returns the shallow tarball as localpath in this case.
Copying it would make the archive differ depending on how the source was
fetched, so unpack git sources like a clone, which uses the shallow tarball.

AI-Generated: Claude Sonnet 5.5
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
 meta/classes/archiver.bbclass            |  9 ++++-
 meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 2 deletions(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 0400d6e373..d1a8dadec7 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -204,10 +204,15 @@ python do_ar_original() {
     fetch = bb.fetch.Fetch(urls, ld)
     tarball_suffix = {}
     for url in fetch.urls:
+        ud = fetch.ud[url]
+        is_git = isinstance(ud.method, bb.fetch.git.Git)
         local = fetch.localpath(url).rstrip("/");
-        if os.path.isfile(local):
+        # The localpath of a git source is the shallow tarball, instead of the
+        # clone, when the fetch used the shallow tarball only. Unpack it like
+        # a clone instead of archiving the tarball itself.
+        if os.path.isfile(local) and not is_git:
             shutil.copy(local, ar_outdir)
-        elif os.path.isdir(local):
+        elif os.path.isdir(local) or (is_git and os.path.isfile(local)):
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
             fetch.unpack(tmpdir, (url,))
             # To handle recipes with more than one source, we add the "name"
diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
index 8c5aa75e9b..84f13839c9 100644
--- a/meta/lib/oeqa/selftest/cases/archiver.py
+++ b/meta/lib/oeqa/selftest/cases/archiver.py
@@ -7,6 +7,8 @@
 import os
 import glob
 import re
+import shutil
+import tarfile
 from oeqa.utils.commands import bitbake, get_bb_vars
 from oeqa.selftest.case import OESelftestTestCase
 
@@ -176,6 +178,52 @@ class Archiver(OESelftestTestCase):
 
         self._test_archiver_mode('original', 'ed-1.21.1.tar.lz')
 
+    def test_archiver_mode_original_git_shallow(self):
+        """
+        Test that the archiver archives a shallow git source with
+        `ARCHIVER_MODE[src] = "original"` when only the shallow tarball is
+        available, but not the clone. This is what happens with the fast
+        initial shallow fetch, or when the shallow tarball is fetched from a
+        mirror.
+        """
+        target = 'gitrepotest'
+
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "original"\n'
+        features += 'BB_GIT_SHALLOW = "1"\n'
+        features += 'BB_GENERATE_SHALLOW_TARBALLS = "1"\n'
+        features += 'DL_DIR = "${TOPDIR}/downloads-shallow"\n'
+        self.write_config(features)
+
+        # Fetch to create the shallow tarball, then remove the clone if the
+        # fetcher fell back to a full clone, so that the second fetch has to
+        # use the shallow tarball. cleansstate makes sure that the archive is
+        # created again and not restored from sstate.
+        bitbake('-c cleansstate %s' % target)
+        bitbake('-c fetch %s' % target)
+        bb_vars = get_bb_vars(['DL_DIR'])
+        self.track_for_cleanup(bb_vars['DL_DIR'])
+        self.assertTrue(glob.glob(os.path.join(bb_vars['DL_DIR'], 'gitshallow_*.tar.gz')),
+                        'Missing shallow tarball in DL_DIR')
+        shutil.rmtree(os.path.join(bb_vars['DL_DIR'], 'git2'))
+        bitbake('-c cleansstate %s' % target)
+
+        bitbake('-c deploy_archives %s' % target)
+
+        names = self._get_original_git_archive_names(target)
+        self.assertTrue([n for n in names if n.endswith('/COPYING')], 'Archive has no source files: %s' % names[:20])
+
+    def _get_original_git_archive_names(self, target):
+        """
+        Return the names of the files in the original source archive of the
+        git source of target.
+        """
+        bb_vars = get_bb_vars(['DEPLOY_DIR_SRC'], target)
+        archives = glob.glob(os.path.join(bb_vars['DEPLOY_DIR_SRC'], '*', '%s-*' % target, '%s-*.tar.*' % target))
+        self.assertEqual(len(archives), 1, 'Expected one source archive for %s, found %s' % (target, archives))
+        with tarfile.open(archives[0]) as tar:
+            return tar.getnames()
+
     def test_archiver_mode_patched(self):
         """
         Test that the archiver works with `ARCHIVER_MODE[src] = "patched"`.
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [PATCH v3 2/3] archiver.bbclass: remove .git from original git source archives
  2026-11-17 12:24 [PATCH v3 0/3] archiver.bbclass: archive shallow git sources which have no clone ecordonnier
  2026-11-17 12:24 ` [PATCH v3 1/3] " ecordonnier
@ 2026-11-17 12:24 ` ecordonnier
  2026-11-17 12:24 ` [PATCH v3 3/3] archiver.bbclass: fail on sources which cannot be archived ecordonnier
  2 siblings, 0 replies; 4+ messages in thread
From: ecordonnier @ 2026-11-17 12:24 UTC (permalink / raw)
  To: openembedded-core; +Cc: Etienne Cordonnier

From: Etienne Cordonnier <ecordonnier@snap.com>

In the original mode, do_ar_original unpacks git sources and archives the
unpacked tree, including its .git. This .git is of no use in the archive:

- For a clone, unpack() uses clone -s, so .git has no objects, only a
  reference to the clone in DL_DIR via objects/info/alternates. It is
  confusing and still wastes a little space.
- For a shallow tarball, .git has all the (compressed) objects, which
  duplicates the sources and doubles the size of the archive.

Remove it, as well as the .git files of submodules. Keep it with nocheckout
(which bareclone implies), where the repository is what gets unpacked.

AI-Generated: Claude Opus 5.5
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
 meta/classes/archiver.bbclass            | 12 +++++++
 meta/lib/oeqa/selftest/cases/archiver.py | 43 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index d1a8dadec7..d22213be33 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -215,6 +215,18 @@ python do_ar_original() {
         elif os.path.isdir(local) or (is_git and os.path.isfile(local)):
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
             fetch.unpack(tmpdir, (url,))
+            if is_git and not ud.nocheckout:
+                # The .git of a clone has no objects (clone -s only references
+                # them via objects/info/alternates), and the one of a shallow
+                # tarball duplicates the sources, so it is of no use in the
+                # archive. This includes the .git files of submodules. Keep it
+                # with nocheckout (and bareclone), where it is the source.
+                for root, dirs, files in os.walk(ud.destdir):
+                    if '.git' in dirs:
+                        dirs.remove('.git')
+                        bb.utils.remove(os.path.join(root, '.git'), recurse=True)
+                    if '.git' in files:
+                        os.remove(os.path.join(root, '.git'))
             # To handle recipes with more than one source, we add the "name"
             # URL parameter as suffix. We treat it as an error when
             # there's more than one URL without a name, or a name gets reused.
diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
index 84f13839c9..d55787cfc4 100644
--- a/meta/lib/oeqa/selftest/cases/archiver.py
+++ b/meta/lib/oeqa/selftest/cases/archiver.py
@@ -212,6 +212,49 @@ class Archiver(OESelftestTestCase):
 
         names = self._get_original_git_archive_names(target)
         self.assertTrue([n for n in names if n.endswith('/COPYING')], 'Archive has no source files: %s' % names[:20])
+        self._assert_no_git(names)
+
+    def test_archiver_mode_original_git(self):
+        """
+        Test that the archive of a git source with
+        `ARCHIVER_MODE[src] = "original"` has no .git directory.
+        """
+        target = 'gitrepotest'
+
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "original"\n'
+        self.write_config(features)
+
+        bitbake('-c cleansstate %s' % target)
+        bitbake('-c deploy_archives %s' % target)
+
+        names = self._get_original_git_archive_names(target)
+        self.assertTrue([n for n in names if n.endswith('/COPYING')], 'Archive has no source files: %s' % names[:20])
+        self._assert_no_git(names)
+
+    def test_archiver_mode_original_gitsm(self):
+        """
+        Test that the archive of a git source with submodules with
+        `ARCHIVER_MODE[src] = "original"` contains the submodules, but neither
+        the .git directory nor the .git files of the submodules.
+        """
+        target = 'git-submodule-test'
+
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "original"\n'
+        features += 'COPYLEFT_LICENSE_INCLUDE = "*"\n'
+        self.write_config(features)
+
+        bitbake('-c cleansstate %s' % target)
+        bitbake('-c deploy_archives %s' % target)
+
+        names = self._get_original_git_archive_names(target)
+        self.assertTrue([n for n in names if n.endswith('/.gitmodules')], 'Archive has no submodules: %s' % names[:20])
+        self._assert_no_git(names)
+
+    def _assert_no_git(self, names):
+        git = [n for n in names if '.git' in n.split('/')]
+        self.assertFalse(git, 'Archive contains .git: %s' % git[:20])
 
     def _get_original_git_archive_names(self, target):
         """
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [PATCH v3 3/3] archiver.bbclass: fail on sources which cannot be archived
  2026-11-17 12:24 [PATCH v3 0/3] archiver.bbclass: archive shallow git sources which have no clone ecordonnier
  2026-11-17 12:24 ` [PATCH v3 1/3] " ecordonnier
  2026-11-17 12:24 ` [PATCH v3 2/3] archiver.bbclass: remove .git from original git source archives ecordonnier
@ 2026-11-17 12:24 ` ecordonnier
  2 siblings, 0 replies; 4+ messages in thread
From: ecordonnier @ 2026-11-17 12:24 UTC (permalink / raw)
  To: openembedded-core; +Cc: Etienne Cordonnier

From: Etienne Cordonnier <ecordonnier@snap.com>

In the original mode, do_ar_original archives a source if the fetcher's
localpath is a file, or a directory. Any other source is silently skipped,
and the source archive is incomplete without any notice. This used to happen
for shallow git fetches without a clone, which is fixed by commit
"archiver.bbclass: archive shallow git sources which have no clone".

No fetcher has a valid case for a localpath which is neither a file nor a
directory after do_fetch, so make it a fatal error.

AI-Generated: Claude Opus 5.5
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
 meta/classes/archiver.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index d22213be33..a54882eeb8 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -248,6 +248,8 @@ python do_ar_original() {
                         bb.fatal("Cannot determine archive names for original source because 'name=' URL parameter '%s' is used twice. Make it unique in: %s %s" % (name, tarball_suffix[name], url))
             tarball_suffix[name] = url
             create_tarball(d, tmpdir + '/.', name, ar_outdir)
+        else:
+            bb.fatal("Cannot archive %s: its local path %s is neither a file nor a directory" % (url, local))
 
     # Emit patch series files for 'original'
     bb.note('Writing patch series files...')
-- 
2.43.0


^ permalink raw reply related	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-17 12:25 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 12:24 [PATCH v3 0/3] archiver.bbclass: archive shallow git sources which have no clone ecordonnier
2026-11-17 12:24 ` [PATCH v3 1/3] " ecordonnier
2026-11-17 12:24 ` [PATCH v3 2/3] archiver.bbclass: remove .git from original git source archives ecordonnier
2026-11-17 12:24 ` [PATCH v3 3/3] archiver.bbclass: fail on sources which cannot be archived ecordonnier

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox