[PATCH v2 0/3] *** archiver.bbclass: archive shallow git sources which have no clone ***

[PATCH v2 0/3] *** archiver.bbclass: archive shallow git sources which have no clone ***

[ecordonnier]

From: Etienne Cordonnier <ecordonnier@snap.com>

*** BLURB HERE ***

Etienne Cordonnier (3):
  archiver.bbclass: archive shallow git sources which have no clone
  archiver.bbclass: remove .git from original git source archives
  archiver.bbclass: fail on sources which cannot be archived

v2:
  - Combined the two v1 patches into one series, as requested.
  - Split "archive shallow git sources which have no clone" into 1/3 (archive the shallow source) and 2/3 (remove .git), and 2/3 now removes .git for all git sources, including the .git files of submodules (kept for nocheckout/bareclone).
  - 3/3 ("warn about sources which are not archived" in v1) now raises a fatal error.

 meta/classes/archiver.bbclass            | 22 +++++-
 meta/lib/oeqa/selftest/cases/archiver.py | 91 ++++++++++++++++++++++++
 2 files changed, 112 insertions(+), 1 deletion(-)

-- 
2.43.0

[PATCH v2 1/3] archiver.bbclass: archive shallow git sources which have no clone

[ecordonnier]

From: Etienne Cordonnier <ecordonnier@snap.com>

In the original mode, do_ar_original archives a source if the fetcher's
localpath is a file, or a directory (which is unpacked and tarred). For git,
localpath is the clone in DL_DIR/git2. A shallow fetch (BB_GIT_SHALLOW) can
use the gitshallow tarball (ud.fullshallow) without creating the clone: since
bitbake commit 457288b2f ("fetch2/git: Add support for fast initial shallow
fetch") this is what a shallow fetch does by default, and it also happens when
the shallow tarball is fetched from a mirror. In this case, the source was
silently skipped and not archived at all.

Treat this case like a directory source, as unpack() populates the
directory from the shallow tarball. Use the same check as unpack()
(clonedir_need_update()), so that the case of an out of date clone, where
unpack() also uses the shallow tarball, is handled as well.

AI-Generated: Claude Sonnet 5.5
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
 meta/classes/archiver.bbclass            |  8 +++-
 meta/lib/oeqa/selftest/cases/archiver.py | 48 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+), 1 deletion(-)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index 0400d6e373..f6cca35186 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -204,10 +204,16 @@ python do_ar_original() {
     fetch = bb.fetch.Fetch(urls, ld)
     tarball_suffix = {}
     for url in fetch.urls:
+        ud = fetch.ud[url]
         local = fetch.localpath(url).rstrip("/");
+        # A shallow git fetch which used the shallow tarball has no up to date
+        # clone, which is what localpath() points to. unpack() then uses the
+        # shallow tarball, using the same check as here.
+        shallow = getattr(ud, 'shallow', False) and os.path.isfile(ud.fullshallow) \
+                  and ud.method.clonedir_need_update(ud, ld)
         if os.path.isfile(local):
             shutil.copy(local, ar_outdir)
-        elif os.path.isdir(local):
+        elif os.path.isdir(local) or shallow:
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
             fetch.unpack(tmpdir, (url,))
             # To handle recipes with more than one source, we add the "name"
diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
index 8c5aa75e9b..84f13839c9 100644
--- a/meta/lib/oeqa/selftest/cases/archiver.py
+++ b/meta/lib/oeqa/selftest/cases/archiver.py
@@ -7,6 +7,8 @@
 import os
 import glob
 import re
+import shutil
+import tarfile
 from oeqa.utils.commands import bitbake, get_bb_vars
 from oeqa.selftest.case import OESelftestTestCase
 
@@ -176,6 +178,52 @@ class Archiver(OESelftestTestCase):
 
         self._test_archiver_mode('original', 'ed-1.21.1.tar.lz')
 
+    def test_archiver_mode_original_git_shallow(self):
+        """
+        Test that the archiver archives a shallow git source with
+        `ARCHIVER_MODE[src] = "original"` when only the shallow tarball is
+        available, but not the clone. This is what happens with the fast
+        initial shallow fetch, or when the shallow tarball is fetched from a
+        mirror.
+        """
+        target = 'gitrepotest'
+
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "original"\n'
+        features += 'BB_GIT_SHALLOW = "1"\n'
+        features += 'BB_GENERATE_SHALLOW_TARBALLS = "1"\n'
+        features += 'DL_DIR = "${TOPDIR}/downloads-shallow"\n'
+        self.write_config(features)
+
+        # Fetch to create the shallow tarball, then remove the clone if the
+        # fetcher fell back to a full clone, so that the second fetch has to
+        # use the shallow tarball. cleansstate makes sure that the archive is
+        # created again and not restored from sstate.
+        bitbake('-c cleansstate %s' % target)
+        bitbake('-c fetch %s' % target)
+        bb_vars = get_bb_vars(['DL_DIR'])
+        self.track_for_cleanup(bb_vars['DL_DIR'])
+        self.assertTrue(glob.glob(os.path.join(bb_vars['DL_DIR'], 'gitshallow_*.tar.gz')),
+                        'Missing shallow tarball in DL_DIR')
+        shutil.rmtree(os.path.join(bb_vars['DL_DIR'], 'git2'))
+        bitbake('-c cleansstate %s' % target)
+
+        bitbake('-c deploy_archives %s' % target)
+
+        names = self._get_original_git_archive_names(target)
+        self.assertTrue([n for n in names if n.endswith('/COPYING')], 'Archive has no source files: %s' % names[:20])
+
+    def _get_original_git_archive_names(self, target):
+        """
+        Return the names of the files in the original source archive of the
+        git source of target.
+        """
+        bb_vars = get_bb_vars(['DEPLOY_DIR_SRC'], target)
+        archives = glob.glob(os.path.join(bb_vars['DEPLOY_DIR_SRC'], '*', '%s-*' % target, '%s-*.tar.*' % target))
+        self.assertEqual(len(archives), 1, 'Expected one source archive for %s, found %s' % (target, archives))
+        with tarfile.open(archives[0]) as tar:
+            return tar.getnames()
+
     def test_archiver_mode_patched(self):
         """
         Test that the archiver works with `ARCHIVER_MODE[src] = "patched"`.
-- 
2.43.0

[PATCH v2 2/3] archiver.bbclass: remove .git from original git source archives

[ecordonnier]

From: Etienne Cordonnier <ecordonnier@snap.com>

In the original mode, do_ar_original unpacks git sources and archives the
unpacked tree, including its .git. This .git is of no use in the archive:

- For a clone, unpack() uses clone -s, so .git has no objects, only a
  reference to the clone in DL_DIR via objects/info/alternates. It is
  confusing and still wastes a little space.
- For a shallow tarball, .git has all the (compressed) objects, which
  duplicates the sources and doubles the size of the archive.

Remove it, as well as the .git files of submodules. Keep it with nocheckout
(which bareclone implies), where the repository is what gets unpacked.

AI-Generated: Claude Opus 5.5
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
 meta/classes/archiver.bbclass            | 12 +++++++
 meta/lib/oeqa/selftest/cases/archiver.py | 43 ++++++++++++++++++++++++
 2 files changed, 55 insertions(+)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index f6cca35186..a6b5706abd 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -216,6 +216,18 @@ python do_ar_original() {
         elif os.path.isdir(local) or shallow:
             tmpdir = tempfile.mkdtemp(dir=d.getVar('ARCHIVER_WORKDIR'))
             fetch.unpack(tmpdir, (url,))
+            if isinstance(ud.method, bb.fetch.git.Git) and not ud.nocheckout:
+                # The .git of a clone has no objects (clone -s only references
+                # them via objects/info/alternates), and the one of a shallow
+                # tarball duplicates the sources, so it is of no use in the
+                # archive. This includes the .git files of submodules. Keep it
+                # with nocheckout (and bareclone), where it is the source.
+                for root, dirs, files in os.walk(ud.destdir):
+                    if '.git' in dirs:
+                        dirs.remove('.git')
+                        bb.utils.remove(os.path.join(root, '.git'), recurse=True)
+                    if '.git' in files:
+                        os.remove(os.path.join(root, '.git'))
             # To handle recipes with more than one source, we add the "name"
             # URL parameter as suffix. We treat it as an error when
             # there's more than one URL without a name, or a name gets reused.
diff --git a/meta/lib/oeqa/selftest/cases/archiver.py b/meta/lib/oeqa/selftest/cases/archiver.py
index 84f13839c9..d55787cfc4 100644
--- a/meta/lib/oeqa/selftest/cases/archiver.py
+++ b/meta/lib/oeqa/selftest/cases/archiver.py
@@ -212,6 +212,49 @@ class Archiver(OESelftestTestCase):
 
         names = self._get_original_git_archive_names(target)
         self.assertTrue([n for n in names if n.endswith('/COPYING')], 'Archive has no source files: %s' % names[:20])
+        self._assert_no_git(names)
+
+    def test_archiver_mode_original_git(self):
+        """
+        Test that the archive of a git source with
+        `ARCHIVER_MODE[src] = "original"` has no .git directory.
+        """
+        target = 'gitrepotest'
+
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "original"\n'
+        self.write_config(features)
+
+        bitbake('-c cleansstate %s' % target)
+        bitbake('-c deploy_archives %s' % target)
+
+        names = self._get_original_git_archive_names(target)
+        self.assertTrue([n for n in names if n.endswith('/COPYING')], 'Archive has no source files: %s' % names[:20])
+        self._assert_no_git(names)
+
+    def test_archiver_mode_original_gitsm(self):
+        """
+        Test that the archive of a git source with submodules with
+        `ARCHIVER_MODE[src] = "original"` contains the submodules, but neither
+        the .git directory nor the .git files of the submodules.
+        """
+        target = 'git-submodule-test'
+
+        features = 'INHERIT += "archiver"\n'
+        features += 'ARCHIVER_MODE[src] = "original"\n'
+        features += 'COPYLEFT_LICENSE_INCLUDE = "*"\n'
+        self.write_config(features)
+
+        bitbake('-c cleansstate %s' % target)
+        bitbake('-c deploy_archives %s' % target)
+
+        names = self._get_original_git_archive_names(target)
+        self.assertTrue([n for n in names if n.endswith('/.gitmodules')], 'Archive has no submodules: %s' % names[:20])
+        self._assert_no_git(names)
+
+    def _assert_no_git(self, names):
+        git = [n for n in names if '.git' in n.split('/')]
+        self.assertFalse(git, 'Archive contains .git: %s' % git[:20])
 
     def _get_original_git_archive_names(self, target):
         """
-- 
2.43.0

[PATCH v2 3/3] archiver.bbclass: fail on sources which cannot be archived

[ecordonnier]

From: Etienne Cordonnier <ecordonnier@snap.com>

In the original mode, do_ar_original archives a source if the fetcher's
localpath is a file, or a directory. Any other source is silently skipped,
and the source archive is incomplete without any notice. Until commit
"archiver.bbclass: archive shallow git sources which have no clone", this
happened for shallow git fetches, whose localpath is the clone in
DL_DIR/git2, which is not created when the shallow tarball is used.

No fetcher has a valid case for a localpath which is neither a file nor a
directory after do_fetch, so make it a fatal error.

AI-Generated: Claude Opus 5.5
Signed-off-by: Etienne Cordonnier <ecordonnier@snap.com>
---
 meta/classes/archiver.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/archiver.bbclass b/meta/classes/archiver.bbclass
index a6b5706abd..ec62b89400 100644
--- a/meta/classes/archiver.bbclass
+++ b/meta/classes/archiver.bbclass
@@ -249,6 +249,8 @@ python do_ar_original() {
                         bb.fatal("Cannot determine archive names for original source because 'name=' URL parameter '%s' is used twice. Make it unique in: %s %s" % (name, tarball_suffix[name], url))
             tarball_suffix[name] = url
             create_tarball(d, tmpdir + '/.', name, ar_outdir)
+        else:
+            bb.fatal("Cannot archive %s: its local path %s is neither a file nor a directory" % (url, local))
 
     # Emit patch series files for 'original'
     bb.note('Writing patch series files...')
-- 
2.43.0