public inbox for openembedded-core@lists.openembedded.org
 help / color / mirror / Atom feed
From: Akash Hadke <akash.hadke27@gmail.com>
To: openembedded-core@lists.openembedded.org
Cc: Philip Lorenz <philip.lorenz@bmw.de>,
	Richard Purdie <richard.purdie@linuxfoundation.org>
Subject: [poky][scarthgap][PATCH 16/23] bitbake: fetch2: Use git-lfs fetch to download objects
Date: Fri,  8 Aug 2025 14:19:24 +0530	[thread overview]
Message-ID: <20250808084931.2156763-16-akash.hadke27@gmail.com> (raw)
In-Reply-To: <20250808084931.2156763-1-akash.hadke27@gmail.com>

From: Philip Lorenz <philip.lorenz@bmw.de>

Its not clear which exact git-lfs versions failed to work with bare
repositories, however git-lfs 2.13.2 which is shipped by Debian 10 (i.e.
the oldest supported distribution by scarthgap) shows no issue when
fetching into a bare repository. Switch to git-lfs fetch which in turn
eliminates issues seen when using the gitsm fetcher with submodules
utilizing lfs.

In these scenarios, fetching of LFS objects did not actually happen as
the gitsm fetcher parametrizes the to be fetched repositories with
`bareclone=1` which in turn means that the target revision was never
checked out (and therefore no LFS objects were fetched).

(Bitbake rev: 501d3317df5780a5b0e3c2562b26ed276c1dbe90)

Signed-off-by: Philip Lorenz <philip.lorenz@bmw.de>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
(cherry picked from commit 9962d198c82d64a1562844f66ba67aba590fa7e4)
Signed-off-by: Akash Hadke <akash.hadke27@gmail.com>
---
 bitbake/lib/bb/fetch2/git.py | 25 +------------------------
 1 file changed, 1 insertion(+), 24 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index e211f7230e..2b6e49362f 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -506,30 +506,7 @@ class Git(FetchMethod):
                 raise bb.fetch2.FetchError("Unable to find revision %s even from upstream" % missing_rev)
 
         if self.lfs_need_update(ud, d):
-            # Unpack temporary working copy, use it to run 'git checkout' to force pre-fetching
-            # of all LFS blobs needed at the srcrev.
-            #
-            # It would be nice to just do this inline here by running 'git-lfs fetch'
-            # on the bare clonedir, but that operation requires a working copy on some
-            # releases of Git LFS.
-            with tempfile.TemporaryDirectory(dir=d.getVar('DL_DIR')) as tmpdir:
-                # Do the checkout. This implicitly involves a Git LFS fetch.
-                Git.unpack(self, ud, tmpdir, d)
-
-                # Scoop up a copy of any stuff that Git LFS downloaded. Merge them into
-                # the bare clonedir.
-                #
-                # As this procedure is invoked repeatedly on incremental fetches as
-                # a recipe's SRCREV is bumped throughout its lifetime, this will
-                # result in a gradual accumulation of LFS blobs in <ud.clonedir>/lfs
-                # corresponding to all the blobs reachable from the different revs
-                # fetched across time.
-                #
-                # Only do this if the unpack resulted in a .git/lfs directory being
-                # created; this only happens if at least one blob needed to be
-                # downloaded.
-                if os.path.exists(os.path.join(ud.destdir, ".git", "lfs")):
-                    runfetchcmd("tar -cf - lfs | tar -xf - -C %s" % ud.clonedir, d, workdir="%s/.git" % ud.destdir)
+            self.lfs_fetch(ud, d, ud.clonedir, ud.revision)
 
     def lfs_fetch(self, ud, d, clonedir, revision, fetchall=False, progresshandler=None):
         """Helper method for fetching Git LFS data"""
-- 
2.25.1



  parent reply	other threads:[~2025-08-08  8:50 UTC|newest]

Thread overview: 48+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2025-08-08  8:49 [poky][scarthgap][PATCH 01/23] bitbake: fetch2/git: Use git shallow fetch to implement clone_shallow_local() Akash Hadke
2025-08-08  8:49 ` [poky][scarthgap][PATCH 02/23] bitbake: bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local() Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 03/23] bitbake: fetch2/git: Enforce default remote name to "origin" Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 04/23] bitbake: gitsm: Add clean function Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 05/23] bitbake: git: Clean shallow mirror tarball Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 06/23] bitbake: git: Clean broken symlink Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 07/23] bitbake: lib: Remove double imports Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 08/23] bitbake: fetch2/git: Add support for fast initial shallow fetch Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 09/23] bitbake: fetch2/gitsm: Unpack even when `ud.clonedir` is not available Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 10/23] bitbake: tests/fetch: Adapt test cases for fast shallow fetches Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 11/23] bitbake: fetch2/git: Restore escape quoting for the git url when used Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 12/23] bitbake: fetch/git: always fetch lfs when creating shallow tarball Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 13/23] bitbake: tests/fetch: Move commonly used imports to top Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 14/23] bitbake: fetch2: Check for git-lfs existence before using it Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 15/23] bitbake: fetch2: Simplify git LFS detection Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` Akash Hadke [this message]
2025-08-08  9:10   ` Patchtest results for [poky][scarthgap][PATCH 16/23] bitbake: fetch2: Use git-lfs fetch to download objects patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 17/23] bitbake: fetch2: Fix incorrect lfs parametrization for submodules Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 18/23] bitbake: fetch2: Fix LFS object checkout in submodules Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 19/23] bitbake: tests/fetch: Test gitsm with LFS Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 20/23] bitbake: fetch2/git: fix shallow clone for tag containing slash Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 21/23] bitbake: fetch2: Move the `ensure_symlink()` function into the `FetchMethod` class Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 22/23] bitbake: fetch2: Ensure a valid symlink in `PREMIRRORS` case when using shallow cloning Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-08  8:49 ` [poky][scarthgap][PATCH 23/23] bitbake: fetch2/git: Add multiple revision support Akash Hadke
2025-08-08  9:10   ` Patchtest results for " patchtest
2025-08-11  8:23   ` [OE-core] " Alexander Kanavin
     [not found]     ` <10458.1760096658331781386@lists.openembedded.org>
2025-10-10 12:41       ` Richard Purdie
2025-08-08  9:10 ` Patchtest results for [poky][scarthgap][PATCH 01/23] bitbake: fetch2/git: Use git shallow fetch to implement clone_shallow_local() patchtest

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20250808084931.2156763-16-akash.hadke27@gmail.com \
    --to=akash.hadke27@gmail.com \
    --cc=openembedded-core@lists.openembedded.org \
    --cc=philip.lorenz@bmw.de \
    --cc=richard.purdie@linuxfoundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox