[PATCH 0/2] fetch2/git: Use git shallow fetch to implement clone_shallow_local()

[PATCH 0/2] fetch2/git: Use git shallow fetch to implement clone_shallow_local()

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

* Test info
  bitbake-selftest works well
  bitbake world works well with the shallow tarballs

// Robert

The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0:

  bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes (2024-06-18 08:45:22 +0100)

are available in the Git repository at:

  https://github.com/robertlinux/yocto rbt/shallow
  https://github.com/robertlinux/yocto/tree/rbt/shallow

Robert Yang (2):
  fetch2/git: Use git shallow fetch to implement clone_shallow_local()
  bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local()

 bitbake/lib/bb/fetch2/git.py  | 71 ++++++++++++++++++++++-------------
 bitbake/lib/bb/tests/fetch.py | 13 ++++---
 2 files changed, 53 insertions(+), 31 deletions(-)

-- 
2.45.1

[PATCH 1/2] fetch2/git: Use git shallow fetch to implement clone_shallow_local()

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

This patch can make the following settings much more faster:
BB_GIT_SHALLOW = "1"
BB_GENERATE_MIRROR_TARBALLS = "1"

* The previous implementation was:
  - Make a full clone for the repo from local ud.clonedir
  - Use git-make-shallow to remove unneeded revs

  It was very slow for recipes which have a lot of SRC_URIs, for example
  vulkan-samples and docker-compose, the docker-compose can't be done after 5
  hours.

  $ bitbake vulkan-samples -cfetch
  Before: 12 minutes
  Now: 2 minutes

  $ bitbake docker-compose -cfetch
  Before: More than 300 minutes
  Now: 15 minutes

* The patch uses git shallow fetch to fetch the repo from local
  ud.clonedir:
  - For BB_GIT_SHALLOW_DEPTH: git fetch --depth <depth> rev
  - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev

  Then the git repo will be shallow, and git-make-shallow is not needed any
  more.

  And git shallow fetch will download less commits than before since it doesn't
  need "rev^" to parse the dependencies, the previous code always need 'rev^'.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/fetch2/git.py | 71 +++++++++++++++++++++++-------------
 1 file changed, 45 insertions(+), 26 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index c7ff769fdf..d7583f0c3a 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -551,18 +551,31 @@ class Git(FetchMethod):
             runfetchcmd("touch %s.done" % ud.fullmirror, d)
 
     def clone_shallow_local(self, ud, dest, d):
-        """Clone the repo and make it shallow.
+        """
+        Shallow fetch from ud.clonedir (${DL_DIR}/git2/<gitrepo> by default):
+        - For BB_GIT_SHALLOW_DEPTH: git fetch --depth <depth> rev
+        - For BB_GIT_SHALLOW_REVS: git fetch --shallow-exclude=<revs> rev
+        """
+
+        bb.utils.mkdirhier(dest)
+        init_cmd = "%s init -q" % ud.basecmd
+        if ud.bareclone:
+            init_cmd += " --bare"
+        runfetchcmd(init_cmd, d, workdir=dest)
+        runfetchcmd("%s remote add origin %s" % (ud.basecmd, ud.clonedir), d, workdir=dest)
 
-        The upstream url of the new clone isn't set at this time, as it'll be
-        set correctly when unpacked."""
-        runfetchcmd("%s clone %s %s %s" % (ud.basecmd, ud.cloneflags, ud.clonedir, dest), d)
+        # Check the histories which should be excluded
+        shallow_exclude = ''
+        for revision in ud.shallow_revs:
+            shallow_exclude += " --shallow-exclude=%s" % revision
 
-        to_parse, shallow_branches = [], []
         for name in ud.names:
             revision = ud.revisions[name]
             depth = ud.shallow_depths[name]
-            if depth:
-                to_parse.append('%s~%d^{}' % (revision, depth - 1))
+
+            # The --depth and --shallow-exclude can't be used together
+            if depth and shallow_exclude:
+                raise bb.fetch2.FetchError("BB_GIT_SHALLOW_REVS is set, but BB_GIT_SHALLOW_DEPTH is not 0.")
 
             # For nobranch, we need a ref, otherwise the commits will be
             # removed, and for non-nobranch, we truncate the branch to our
@@ -575,36 +588,42 @@ class Git(FetchMethod):
             else:
                 ref = "refs/remotes/origin/%s" % branch
 
-            shallow_branches.append(ref)
-            runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest)
+            fetch_cmd = "%s fetch origin %s" % (ud.basecmd, revision)
+            if depth:
+                fetch_cmd += " --depth %s" % depth
 
-        # Map srcrev+depths to revisions
-        parsed_depths = runfetchcmd("%s rev-parse %s" % (ud.basecmd, " ".join(to_parse)), d, workdir=dest)
+            if shallow_exclude:
+                fetch_cmd += shallow_exclude
 
-        # Resolve specified revisions
-        parsed_revs = runfetchcmd("%s rev-parse %s" % (ud.basecmd, " ".join('"%s^{}"' % r for r in ud.shallow_revs)), d, workdir=dest)
-        shallow_revisions = parsed_depths.splitlines() + parsed_revs.splitlines()
+            runfetchcmd(fetch_cmd, d, workdir=dest)
+            runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest)
 
         # Apply extra ref wildcards
-        all_refs = runfetchcmd('%s for-each-ref "--format=%%(refname)"' % ud.basecmd,
-                               d, workdir=dest).splitlines()
+        all_refs_remote = runfetchcmd("%s ls-remote origin 'refs/*'" % ud.basecmd, \
+                                        d, workdir=dest).splitlines()
+        all_refs = []
+        for line in all_refs_remote:
+            all_refs.append(line.split()[-1])
+        extra_refs = []
         for r in ud.shallow_extra_refs:
             if not ud.bareclone:
                 r = r.replace('refs/heads/', 'refs/remotes/origin/')
 
             if '*' in r:
                 matches = filter(lambda a: fnmatch.fnmatchcase(a, r), all_refs)
-                shallow_branches.extend(matches)
+                extra_refs.extend(matches)
             else:
-                shallow_branches.append(r)
-
-        # Make the repository shallow
-        shallow_cmd = [self.make_shallow_path, '-s']
-        for b in shallow_branches:
-            shallow_cmd.append('-r')
-            shallow_cmd.append(b)
-        shallow_cmd.extend(shallow_revisions)
-        runfetchcmd(subprocess.list2cmdline(shallow_cmd), d, workdir=dest)
+                extra_refs.append(r)
+
+        for ref in extra_refs:
+            ref_fetch = os.path.basename(ref)
+            runfetchcmd("%s fetch origin --depth 1 %s" % (ud.basecmd, ref_fetch), d, workdir=dest)
+            revision = runfetchcmd("%s rev-parse FETCH_HEAD" % ud.basecmd, d, workdir=dest)
+            runfetchcmd("%s update-ref %s %s" % (ud.basecmd, ref, revision), d, workdir=dest)
+
+        # The url is local ud.clonedir, set it to upstream one
+        repourl = self._get_repo_url(ud)
+        runfetchcmd("%s remote set-url origin %s" % (ud.basecmd, shlex.quote(repourl)), d, workdir=dest)
 
     def unpack(self, ud, destdir, d):
         """ unpack the downloaded src to destdir"""
-- 
2.45.1

[PATCH 2/2] bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local()

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

Update the test cases since the implementation is changed:

* test_shallow_multi_one_uri()
  The a_branch and v0.0 had the same revision, and it required fetch a_branch
  and remove histories of v0.0 which were conflicted, and bitbake reported:
  fatal: no commits selected for shallow requests

  Make a_branch and v0.0 have different revs to fix the problem.

  And now the 'rev^' is not needed, so update self.assertRevCount() as well.

* test_shallow_multi_one_uri_depths()
  Update self.assertRevCount(), now git only fetches the required revs.

* test_shallow_fetch_missing_revs()
  The command is:
  $ git fetch --shallow-exclude=v0.0 master

  But master and v0.0 uses the same revision, so there is no commit to fetch.

* test_shallow_fetch_missing_revs_fails()
  Two unneeded committs are not fetched now:
  - rev^
  - One not specified or required tag.

  So update self.assertRevCount()

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/tests/fetch.py | 13 ++++++++-----
 1 file changed, 8 insertions(+), 5 deletions(-)

diff --git a/bitbake/lib/bb/tests/fetch.py b/bitbake/lib/bb/tests/fetch.py
index 701129d138..2ef2063436 100644
--- a/bitbake/lib/bb/tests/fetch.py
+++ b/bitbake/lib/bb/tests/fetch.py
@@ -2034,9 +2034,9 @@ class GitShallowTest(FetcherTest):
         self.add_empty_file('b')
         self.git('checkout -b a_branch', cwd=self.srcdir)
         self.add_empty_file('c')
+        self.git('tag v0.0 HEAD', cwd=self.srcdir)
         self.add_empty_file('d')
         self.git('checkout master', cwd=self.srcdir)
-        self.git('tag v0.0 a_branch', cwd=self.srcdir)
         self.add_empty_file('e')
         self.git('merge --no-ff --no-edit a_branch', cwd=self.srcdir)
         self.add_empty_file('f')
@@ -2052,7 +2052,7 @@ class GitShallowTest(FetcherTest):
 
         self.fetch_shallow(uri)
 
-        self.assertRevCount(5)
+        self.assertRevCount(4)
         self.assertRefs(['master', 'origin/master', 'origin/a_branch'])
 
     def test_shallow_multi_one_uri_depths(self):
@@ -2199,7 +2199,7 @@ class GitShallowTest(FetcherTest):
 
         self.fetch_shallow()
 
-        self.assertRevCount(5)
+        self.assertRevCount(2)
 
     def test_shallow_invalid_revs(self):
         self.add_empty_file('a')
@@ -2218,7 +2218,10 @@ class GitShallowTest(FetcherTest):
         self.git('tag v0.0 master', cwd=self.srcdir)
         self.d.setVar('BB_GIT_SHALLOW_DEPTH', '0')
         self.d.setVar('BB_GIT_SHALLOW_REVS', 'v0.0')
-        self.fetch_shallow()
+
+        with self.assertRaises(bb.fetch2.FetchError), self.assertLogs("BitBake.Fetcher", level="ERROR") as cm:
+            self.fetch_shallow()
+        self.assertIn("fatal: no commits selected for shallow requests", cm.output[0])
 
     def test_shallow_fetch_missing_revs_fails(self):
         self.add_empty_file('a')
@@ -2249,7 +2252,7 @@ class GitShallowTest(FetcherTest):
         revs = len(self.git('rev-list master').splitlines())
         self.assertNotEqual(orig_revs, revs)
         self.assertRefs(['master', 'origin/master'])
-        self.assertRevCount(orig_revs - 1758)
+        self.assertRevCount(orig_revs - 1760)
 
     def test_that_unpack_throws_an_error_when_the_git_clone_nor_shallow_tarball_exist(self):
         self.add_empty_file('a')
-- 
2.45.1

Re: [bitbake-devel] [PATCH 0/2] fetch2/git: Use git shallow fetch to implement clone_shallow_local()

[Robert Yang]

Hi RP and Christopher,

On 6/20/24 20:44, Robert Yang via lists.openembedded.org wrote:
> From: Robert Yang <liezhi.yang@windriver.com>
> 
> * Test info
>    bitbake-selftest works well
>    bitbake world works well with the shallow tarballs

Do you have any comments on the patches, please?

// Robert

> 
> // Robert
> 
> The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0:
> 
>    bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes (2024-06-18 08:45:22 +0100)
> 
> are available in the Git repository at:
> 
>    https://github.com/robertlinux/yocto rbt/shallow
>    https://github.com/robertlinux/yocto/tree/rbt/shallow
> 
> Robert Yang (2):
>    fetch2/git: Use git shallow fetch to implement clone_shallow_local()
>    bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local()
> 
>   bitbake/lib/bb/fetch2/git.py  | 71 ++++++++++++++++++++++-------------
>   bitbake/lib/bb/tests/fetch.py | 13 ++++---
>   2 files changed, 53 insertions(+), 31 deletions(-)
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16368): https://lists.openembedded.org/g/bitbake-devel/message/16368
> Mute This Topic: https://lists.openembedded.org/mt/106779045/7304958
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [liezhi.yang@eng.windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [bitbake-devel] [PATCH 0/2] fetch2/git: Use git shallow fetch to implement clone_shallow_local()

[Alexandre Belloni]

Hello,

This fails on our ubuntu 20.04 workers:

https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6923/steps/11/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3542/steps/11/logs/stdio

On 20/06/2024 05:44:39-0700, Robert Yang via lists.openembedded.org wrote:
> From: Robert Yang <liezhi.yang@windriver.com>
> 
> * Test info
>   bitbake-selftest works well
>   bitbake world works well with the shallow tarballs
> 
> // Robert
> 
> The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0:
> 
>   bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes (2024-06-18 08:45:22 +0100)
> 
> are available in the Git repository at:
> 
>   https://github.com/robertlinux/yocto rbt/shallow
>   https://github.com/robertlinux/yocto/tree/rbt/shallow
> 
> Robert Yang (2):
>   fetch2/git: Use git shallow fetch to implement clone_shallow_local()
>   bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local()
> 
>  bitbake/lib/bb/fetch2/git.py  | 71 ++++++++++++++++++++++-------------
>  bitbake/lib/bb/tests/fetch.py | 13 ++++---
>  2 files changed, 53 insertions(+), 31 deletions(-)
> 
> -- 
> 2.45.1
> 

> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16368): https://lists.openembedded.org/g/bitbake-devel/message/16368
> Mute This Topic: https://lists.openembedded.org/mt/106779045/3617179
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [alexandre.belloni@bootlin.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 


-- 
Alexandre Belloni, co-owner and COO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: [bitbake-devel] [PATCH 0/2] fetch2/git: Use git shallow fetch to implement clone_shallow_local()

[Robert Yang]

On 7/2/24 06:13, Alexandre Belloni via lists.openembedded.org wrote:
> Hello,
> 
> This fails on our ubuntu 20.04 workers:
> 
> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6923/steps/11/logs/stdio
> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3542/steps/11/logs/stdio

Sorry, it might because the git version on ubuntu 20.04 is old, let me check how 
to fix it.

// Robert

> 
> On 20/06/2024 05:44:39-0700, Robert Yang via lists.openembedded.org wrote:
>> From: Robert Yang <liezhi.yang@windriver.com>
>>
>> * Test info
>>    bitbake-selftest works well
>>    bitbake world works well with the shallow tarballs
>>
>> // Robert
>>
>> The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0:
>>
>>    bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError also in get_unihashes (2024-06-18 08:45:22 +0100)
>>
>> are available in the Git repository at:
>>
>>    https://github.com/robertlinux/yocto rbt/shallow
>>    https://github.com/robertlinux/yocto/tree/rbt/shallow
>>
>> Robert Yang (2):
>>    fetch2/git: Use git shallow fetch to implement clone_shallow_local()
>>    bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local()
>>
>>   bitbake/lib/bb/fetch2/git.py  | 71 ++++++++++++++++++++++-------------
>>   bitbake/lib/bb/tests/fetch.py | 13 ++++---
>>   2 files changed, 53 insertions(+), 31 deletions(-)
>>
>> -- 
>> 2.45.1
>>
> 
>>
>>
>>
> 
> 
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16383): https://lists.openembedded.org/g/bitbake-devel/message/16383
> Mute This Topic: https://lists.openembedded.org/mt/106779045/3616940
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [liezhi.yang@windriver.com]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [bitbake-devel] [PATCH 0/2] fetch2/git: Use git shallow fetch to implement clone_shallow_local()

[Robert Yang]

On 7/2/24 15:31, Robert Yang wrote:
> 
> 
> On 7/2/24 06:13, Alexandre Belloni via lists.openembedded.org wrote:
>> Hello,
>>
>> This fails on our ubuntu 20.04 workers:
>>
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/87/builds/6923/steps/11/logs/stdio
>> https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3542/steps/11/logs/stdio
> 
> Sorry, it might because the git version on ubuntu 20.04 is old, let me check how  > to fix it.

I had a ppa on my ubuntu host 20.04 which installed a higher version git, that
made me didn't meet the error. I've tried pure ubuntu 20.04 from docker hub, and
then reproduced the error, the fix is:

+            # Advertise the revision for lower version git such as 2.25.1:
+            # error: Server does not allow request for unadvertised object.
+            # The ud.clonedir is a local temporary dir, will be removed when
+            # fetch is done, so we can do anything on it.
+            adv_cmd = 'git branch -f advertise-%s %s' % (revision, revision)
+            runfetchcmd(adv_cmd, d, workdir=ud.clonedir)


I will send a V2 for it.

// Robert

> 
> // Robert
> 
>>
>> On 20/06/2024 05:44:39-0700, Robert Yang via lists.openembedded.org wrote:
>>> From: Robert Yang <liezhi.yang@windriver.com>
>>>
>>> * Test info
>>>    bitbake-selftest works well
>>>    bitbake world works well with the shallow tarballs
>>>
>>> // Robert
>>>
>>> The following changes since commit 5d88faa0f35f0205c1475893d8589d1e6533dcc0:
>>>
>>>    bitbake: siggen: catch FileNotFoundError everywhere and ConnectionError 
>>> also in get_unihashes (2024-06-18 08:45:22 +0100)
>>>
>>> are available in the Git repository at:
>>>
>>>    https://github.com/robertlinux/yocto rbt/shallow
>>>    https://github.com/robertlinux/yocto/tree/rbt/shallow
>>>
>>> Robert Yang (2):
>>>    fetch2/git: Use git shallow fetch to implement clone_shallow_local()
>>>    bitbake: tests/fetch: Update GitShallowTest for clone_shallow_local()
>>>
>>>   bitbake/lib/bb/fetch2/git.py  | 71 ++++++++++++++++++++++-------------
>>>   bitbake/lib/bb/tests/fetch.py | 13 ++++---
>>>   2 files changed, 53 insertions(+), 31 deletions(-)
>>>
>>> -- 
>>> 2.45.1
>>>
>>
>>>
>>>
>>>
>>
>>
>>
>>
>> -=-=-=-=-=-=-=-=-=-=-=-
>> Links: You receive all messages sent to this group.
>> View/Reply Online (#16383): 
>> https://lists.openembedded.org/g/bitbake-devel/message/16383
>> Mute This Topic: https://lists.openembedded.org/mt/106779045/3616940
>> Group Owner: bitbake-devel+owner@lists.openembedded.org
>> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub 
>> [liezhi.yang@windriver.com]
>> -=-=-=-=-=-=-=-=-=-=-=-
>>