[PATCH 0/1] fetch2/git.py: Try without '.git' suffix firstly

[PATCH 0/1] fetch2/git.py: Try without '.git' suffix firstly

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

The following changes since commit 79ab2413e082838291157445038476f9d8edb586:

  cpio: mark CVE-2023-7216 as disputed (2024-05-14 12:43:23 +0100)

are available in the Git repository at:

  https://github.com/robertlinux/yocto rbt/fetch
  https://github.com/robertlinux/yocto/tree/rbt/fetch

Robert Yang (1):
  fetch2/git.py: Try without '.git' suffix firstly

 bitbake/lib/bb/fetch2/git.py | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

-- 
2.35.5

[PATCH 1/1] fetch2/git.py: Try without '.git' suffix firstly

[liezhi.yang]

From: Robert Yang <liezhi.yang@windriver.com>

If the repo on server is foo.git, both of the following commands
will work:
1) $ git clone <url>/foo.git
2) $ git clone <url>/foo
But only the second command works if the repo server is foo
(without .git suffix), so try without '.git' suffix firstly.

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
---
 bitbake/lib/bb/fetch2/git.py | 23 +++++++++++++++++++++--
 1 file changed, 21 insertions(+), 2 deletions(-)

diff --git a/bitbake/lib/bb/fetch2/git.py b/bitbake/lib/bb/fetch2/git.py
index c7ff769fdfe..0134344cdf3 100644
--- a/bitbake/lib/bb/fetch2/git.py
+++ b/bitbake/lib/bb/fetch2/git.py
@@ -442,11 +442,30 @@ class Git(FetchMethod):
                 objects = os.path.join(repourl_path, 'objects')
                 if os.path.isdir(objects) and not os.path.islink(objects):
                     repourl = repourl_path
-            clone_cmd = "LANG=C %s clone --bare --mirror %s %s --progress" % (ud.basecmd, shlex.quote(repourl), ud.clonedir)
+            clone_cmd_pre = "LANG=C %s clone --bare --mirror --progress" % ud.basecmd
+            clone_cmd = "%s %s %s" % (clone_cmd_pre, shlex.quote(repourl), ud.clonedir)
             if ud.proto.lower() != 'file':
                 bb.fetch2.check_network_access(d, clone_cmd, ud.url)
             progresshandler = GitProgressHandler(d)
-            runfetchcmd(clone_cmd, d, log=progresshandler)
+            # If the repo on server is foo.git, both of the following commands
+            # will work:
+            # 1) $ git clone <url>/foo.git
+            # 2) $ git clone <url>/foo
+            # But only the second command works if the repo server is foo
+            # (without .git suffix), so try without '.git' suffix firstly
+            cloned_no_git = False
+            if repourl.endswith('.git'):
+                repourl_no_git = repourl[0:-4]
+                clone_cmd_no_git = "%s %s %s" % (clone_cmd_pre, shlex.quote(repourl_no_git), ud.clonedir)
+                try:
+                    runfetchcmd(clone_cmd_no_git, d, log=progresshandler)
+                    cloned_no_git = True
+                except bb.fetch2.FetchError:
+                    # The error messages have already been saved in the log, so
+                    # just pass to next cmd.
+                    pass
+            if not cloned_no_git:
+                runfetchcmd(clone_cmd, d, log=progresshandler)
 
         # Update the checkout if needed
         if self.clonedir_need_update(ud, d):
-- 
2.35.5

Re: [bitbake-devel] [PATCH 1/1] fetch2/git.py: Try without '.git' suffix firstly

[Quentin Schulz]

Hi Robert,

On 5/15/24 11:26 AM, Robert Yang via lists.openembedded.org wrote:
> From: Robert Yang <liezhi.yang@windriver.com>
> 
> If the repo on server is foo.git, both of the following commands
> will work:
> 1) $ git clone <url>/foo.git
> 2) $ git clone <url>/foo
> But only the second command works if the repo server is foo
> (without .git suffix), so try without '.git' suffix firstly.
> 

Shouldn't the SRC_URI in the recipe be fixed instead of the git fetcher? 
Which use case do we need to support here? Something to do with 
mirroring maybe?

Cheers,
Quentin

Re: [bitbake-devel] [PATCH 1/1] fetch2/git.py: Try without '.git' suffix firstly

[Robert Yang]

Hi Quentin,

On 5/15/24 17:35, Quentin Schulz wrote:
> Hi Robert,
> 
> On 5/15/24 11:26 AM, Robert Yang via lists.openembedded.org wrote:
>> From: Robert Yang <liezhi.yang@windriver.com>
>>
>> If the repo on server is foo.git, both of the following commands
>> will work:
>> 1) $ git clone <url>/foo.git
>> 2) $ git clone <url>/foo
>> But only the second command works if the repo server is foo
>> (without .git suffix), so try without '.git' suffix firstly.
>>
> 
> Shouldn't the SRC_URI in the recipe be fixed instead of the git fetcher? Which 
> use case do we need to support here? Something to do with mirroring maybe
The usage is related to repo command + http/https mirror:

* The foo.git on http/https MIRROR works with do_fetch
* But 'repo' command always add a '.git' suffix to the reponame, the foo.git
will be saved as foo.git.git on local disk, and do_fetch would not work with
the local mirror.

Our solution is rename foo.git to foo on http/https mirror, and apply this patch
to make both http(s) and local mirror work with repo command.

It's a little special to our Wind River Linux, and we need send the patch to
mailing list for Yocto compliance.

// Robert

> 
> Cheers,
> Quentin

Re: [bitbake-devel] [PATCH 1/1] fetch2/git.py: Try without '.git' suffix firstly

[Quentin Schulz]

Hi Robert,

On 5/15/24 11:57 AM, Robert Yang via lists.openembedded.org wrote:
> Hi Quentin,
> 
> On 5/15/24 17:35, Quentin Schulz wrote:
>> Hi Robert,
>>
>> On 5/15/24 11:26 AM, Robert Yang via lists.openembedded.org wrote:
>>> From: Robert Yang <liezhi.yang@windriver.com>
>>>
>>> If the repo on server is foo.git, both of the following commands
>>> will work:
>>> 1) $ git clone <url>/foo.git
>>> 2) $ git clone <url>/foo
>>> But only the second command works if the repo server is foo
>>> (without .git suffix), so try without '.git' suffix firstly.
>>>
>>
>> Shouldn't the SRC_URI in the recipe be fixed instead of the git 
>> fetcher? Which use case do we need to support here? Something to do 
>> with mirroring maybe
> The usage is related to repo command + http/https mirror:
> 
> * The foo.git on http/https MIRROR works with do_fetch
> * But 'repo' command always add a '.git' suffix to the reponame, the 
> foo.git
> will be saved as foo.git.git on local disk, and do_fetch would not work 
> with
> the local mirror.
> 
> Our solution is rename foo.git to foo on http/https mirror, and apply 
> this patch
> to make both http(s) and local mirror work with repo command.
> 
> It's a little special to our Wind River Linux, and we need send the 
> patch to
> mailing list for Yocto compliance.
> 

I'm not sure this is something we should fix in BitBake since it's 
related to the use of some external setup tool? I'll let more 
knowledgeable people discuss this :)

In any case, I would suggest to really provide more info (the one as 
above for example) at the very least in the commit log, and even better 
in-code comment so that we don't remove it the day we want to de-clutter 
code. The issue is that this looks very much like code we don't need, 
without proper explanation of the use case (which you now provided).

Cheers,
Quentin

> // Robert
> 
>>
>> Cheers,
>> Quentin
> 
> 
> -=-=-=-=-=-=-=-=-=-=-=-
> Links: You receive all messages sent to this group.
> View/Reply Online (#16222): https://lists.openembedded.org/g/bitbake-devel/message/16222
> Mute This Topic: https://lists.openembedded.org/mt/106110992/6293953
> Group Owner: bitbake-devel+owner@lists.openembedded.org
> Unsubscribe: https://lists.openembedded.org/g/bitbake-devel/unsub [quentin.schulz@cherry.de]
> -=-=-=-=-=-=-=-=-=-=-=-
> 

Re: [bitbake-devel] [PATCH 1/1] fetch2/git.py: Try without '.git' suffix firstly

[Richard Purdie]

On Wed, 2024-05-15 at 17:57 +0800, Robert Yang via lists.openembedded.org wrote:
> Hi Quentin,
> 
> On 5/15/24 17:35, Quentin Schulz wrote:
> > Hi Robert,
> > 
> > On 5/15/24 11:26 AM, Robert Yang via lists.openembedded.org wrote:
> > > From: Robert Yang <liezhi.yang@windriver.com>
> > > 
> > > If the repo on server is foo.git, both of the following commands
> > > will work:
> > > 1) $ git clone <url>/foo.git
> > > 2) $ git clone <url>/foo
> > > But only the second command works if the repo server is foo
> > > (without .git suffix), so try without '.git' suffix firstly.
> > > 
> > 
> > Shouldn't the SRC_URI in the recipe be fixed instead of the git fetcher? Which 
> > use case do we need to support here? Something to do with mirroring maybe
> The usage is related to repo command + http/https mirror:
> 
> * The foo.git on http/https MIRROR works with do_fetch
> * But 'repo' command always add a '.git' suffix to the reponame, the foo.git
> will be saved as foo.git.git on local disk, and do_fetch would not work with
> the local mirror.
> 
> Our solution is rename foo.git to foo on http/https mirror, and apply this patch
> to make both http(s) and local mirror work with repo command.
> 
> It's a little special to our Wind River Linux, and we need send the patch to
> mailing list for Yocto compliance.

I'm with Quentin, the SRC_URIs should be fixed or this should be in
repo specific codepaths. We're not adding a workaround like that into
the git fetcher.

Cheers,

Richard