Re: [bitbake-devel] [PATCH] [RFC] fetch2/git: Prevent git fetcher from fetching gitlab repository metadata

[Marek Vasut <marex@denx.de>]

On 8/22/22 17:21, Peter Kjellerstedt wrote:
>> -----Original Message-----
>> From: Richard Purdie <richard.purdie@linuxfoundation.org>
>> Sent: den 22 augusti 2022 16:17
>> To: Marek Vasut <marex@denx.de>; Quentin Schulz <quentin.schulz@theobroma-
>> systems.com>; Alexander Kanavin <alex.kanavin@gmail.com>
>> Cc: Mikko Rapeli <Mikko.Rapeli@bmw.de>; Martin Jansa
>> <Martin.Jansa@gmail.com>; bitbake-devel <bitbake-
>> devel@lists.openembedded.org>; Peter Kjellerstedt
>> <peter.kjellerstedt@axis.com>
>> Subject: Re: [bitbake-devel] [PATCH] [RFC] fetch2/git: Prevent git fetcher
>> from fetching gitlab repository metadata
>>
>> On Mon, 2022-08-22 at 13:55 +0200, Marek Vasut wrote:
>>> On 8/22/22 12:57, Quentin Schulz wrote:
>>>> Hi Marek,
>>>>
>>>> On 8/22/22 12:35, Marek Vasut wrote:
>>>>> On 8/22/22 10:41, Alexander Kanavin wrote:
>>>>>> On Mon, 22 Aug 2022 at 10:37, Marek Vasut <marex@denx.de> wrote:
>>>>>>
>>>>>>>> So maybe the easy way out is, if nobranch=1 then fetch
>> everything,
>>>>>>>> else
>>>>>>>> just heads and tags ?
>>>>>>>
>>>>>>> No, this won't do, nobranch expects the commit to be in a tag.
>>>>>>
>>>>>> I don't think it expects that.
>>>>>
>>>>> Documentation says it does:
>>>>>
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-
>> 3A__git.openembedded.org_bitbake_tree_lib_bb_fetch2_git.py-
>> 23n45&d=DwICaQ&c=_sEr5x9kUWhuk4_nFwjJtA&r=LYjLexDn7rXIzVmkNPvw5ymA1XTSqHGq
>> 8yBP6m6qZZ4njZguQhZhkI_-
>> 172IIy1t&m=uZNWVMGEowy_ntO8q5fjINXu3LIe9haqbSTYjwWiqO6Q5sEPsUIx5nw28YTBw6o
>> I&s=_AJ3mkGnnM8peSNP8k6MePZ0RtEkQLo7yS1Cll2yjmc&e= "
>>>>> - nobranch
>>>>>      Don't check the SHA validation for branch. set this option for
>> the
>>>>> recipe
>>>>>      referring to commit which is valid in tag instead of branch.
>>>>
>>>> I assume this was meant to give the example of tags which aren't
>>>> necessarily in a branch (annotated tags or tags of commits not belong
>> to
>>>> any branch anymore (force-push for example, or branch deletion).
>>>>
>>>> The git fetcher does a git log --pretty=oneline -n 1 <hash> when
>>>> nobranch is set, otherwise git branch --contains <hash> --list
>> <branch>
>>>> to check whether a commit exists and can be used by bitbake.
>>>>
>>>> Considering this check, I assume nobranch=1 is working for any commit
>>>> that was fetched by the git fetcher?
>>>>
>>>> (We need to update the docs to reflect that in that case).
>>>
>>> In that case, 'git fetch refs/*' in case nobranch is set and 'refs/head
>>> refs/tags' otherwise ?
>>
>> This does get a bit more complex though since you now need two
>> different mirror tarballs, one for each option. The code can do that if
>> setup correctly but we do need to cover that issue.
>>
>> Cheers,
>>
>> Richard
> 
> I made some testing, and for Gerrit to continue to work it would be
> enough to use:
> 
>              fetch_cmd = "LANG=C %s fetch -f --progress %s refs/heads/*:refs/heads/* refs/tags/*:refs/tags/* refs/changes/*:refs/changes/*" % (ud.basecmd, shlex.quote(repourl))
> 
> This should not affect other Git servers and should avoid using
> different fetch commands depending on the URL. The drawback is of
> course that for Gerrit, there would be only marginal benefits to
> this change since the majority of its metadata is in the
> refs/changes space.
> 
> However, I wonder if the suggested change actually has any significant
> effect, given that the initial clone is done using --mirror, which means
> all refs/ spaces are fetched. If I remove the --mirror option from the
> clone command the change works as expected, but I have no idea if that
> has any other significant impact...

With this change, I am able to actually fetch mesa from 
gitlab.freedesktop.org without local CI proxy terminating the connection 
in the process. So yes, it does have effect.