Re: [PATCH v3 1/4] fetch2/git: Add support for fast initial shallow fetch

[Richard Purdie <richard.purdie@linuxfoundation.org>]

On Fri, 2025-02-21 at 17:18 +0000, Koch, Stefan wrote:
> On Fri, 2025-02-21 at 17:11 +0000, Richard Purdie wrote:
> > On Fri, 2025-02-21 at 16:40 +0000, Koch, Stefan wrote:
> > > On Fri, 2025-02-21 at 08:09 +0000, Richard Purdie wrote:
> > > > On Thu, 2025-02-20 at 18:27 +0100, Stefan Koch wrote:
> > > > > When `ud.shallow == 1`:
> > > > > - Prefer an initial shallow clone over an initial full bare
> > > > > clone,
> > > > >   while still utilizing any already existing full bare clones.
> > > > > 
> > > > > This improves:
> > > > > - Resolve timeout issues during initial clones on slow internet
> > > > > connections
> > > > >   by reducing the amount of data transferred.
> > > > > - Eliminate the need to use an HTTPS tarball `SRC_URI`
> > > > >   to reduce data transfer.
> > > > > - Allow SSH-based authentication (e.g. cert and agent-based)
> > > > > when
> > > > >   using non-public repos, so additional HTTPS tokens may not be
> > > > > required.
> > > > > 
> > > > > Signed-off-by: Stefan Koch <stefan-koch@siemens.com>
> > > > > ---
> > > > >  lib/bb/fetch2/git.py | 100 ++++++++++++++++++++++++++++++++---
> > > > > --
> > > > > --
> > > > > ----
> > > > >  1 file changed, 74 insertions(+), 26 deletions(-)
> > > > 
> > > > In testing it looks like this series has some issue on Ubuntu
> > > > 20.04
> > > > systems (both x86_64 and arm):
> > > > 
> > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/1097
> > > > https://autobuilder.yoctoproject.org/valkyrie/#/builders/54/builds/1016
> > > > 
> > > > This is probably due to the version of git there...
> > > 
> > > This issue could occur in two cases:
> > > a) old Git client version
> > > this could be relevant:
> > > https://lore.kernel.org/git/1630496467881.2030439049.4247253551@vivaldi.com/T/
> > 
> > I suspect the git version is the cause of the issue.
> > 
> > > b) Git server version, that has disabled announcements
> > > see:
> > > https://superuser.com/questions/1342547/unable-to-fetch-particular-commit-hash-from-gerit-error-server-does-not-allow-r
> > 
> > We're testing against the same server and we see some passes and only
> > failures on Ubuntu 20.04 so it is unlikely to be this.
> > 
> > > So, I have reintroduced the switch from v2 patch with inverse
> > > logic.
> > > 1. enabled by default
> > > 2. old behaviour can be forced by variable
> > 
> > So in CI on the autobuilder, we need to force the variable to allow
> > the
> > selftests to pass on 20.04 hosts? We also need to error if anyone
> > enables that form of fetching on 20.04 hosts too?
> 
> My current v4 implementation does it automatically in failure case.
> Because the disabled server side issue is possible at end users but not
> BB CI/CD (there only 20.04)

Ok, automatic fallback sounds ok as long as it doesn't complicate the
code too much. I'm not sure we want to have a variable for it though, I
don't like users needing to care about setting such things from a
usability standpoint.

> > > 3. old behaviour is triggered automatically in mentioned failure
> > > case
> > > (see above)
> > > 
> > > I'll send updated v4 patch series.
> This patch should work with 20.04, too.
> > > 
> > 
> > I'm afraid this solution won't work for us. We either need to find a
> > work around, fall back to a different way of fetching or I guess we
> > have to drop support for Ubuntu 20.04 without using builtools tarball
> > and bump our minimum git version requirements.
> Drop 20.04 might be possible to. The EOL of 20.04 comes in a few
> months...
> 
> I would prefer the automatic error handling, to support old git server
> versions at end users, too.

There are other reasons we've wondered about using buildtools on 20.04
but I would prefer not to so this seems reasonable assuming the patch
isn't too bad...

Cheers,

Richard