ls-refs protocol extension

ls-refs protocol extension

[Christian Schilling]

Hi list,
I have been working on a custom git server where the sha values
required to respond to a ls-refs command can occasionally be expensive
to compute.
It would be a great improvement of user experience if it was possible
to show progress to the user while this is happening.
I was hoping that this could be solved by using the already existing
sideband-all capability, but unfortunately it seems that even with
sideband-all the sideband is not available during ls-refs.
So my plan would be to start working on a patch to add sideband
support to ls-refs, unless there is
some other protocol feature that I have overlooked and that would
enable my use case.

Any pointers to either alternative solutions and/or hints to potential
pitfalls in the implementation
of my suggestion would be greatly appreciated.

Best regards,
Christian Schilling

Re: ls-refs protocol extension

[Taylor Blau]

On Sun, Jan 24, 2021 at 01:40:08PM +0100, Christian Schilling wrote:
> Hi list,
> I have been working on a custom git server where the sha values
> required to respond to a ls-refs command can occasionally be expensive
> to compute.

Almost certainly the expensive part of ls-refs is actually traversing
the references, not parsing the objects they point at and determining
their hash.

Incidentally, we had a discussion recently [1] that resulted in some
patches that make it so that ls-refs often only has to read through part
of the refs in your repository, not all of them.

> It would be a great improvement of user experience if it was possible
> to show progress to the user while this is happening.

It's possible that that might help, but honestly I'd be surprised if
there was a real use-case that needed it (especially after the patches
that I mentioned which should make it fast enough that you don't have to
care :-)).

Thanks,
Taylor

[1]: https://lore.kernel.org/git/20210119144251.27924-1-jacob@gitlab.com/

Re: ls-refs protocol extension

[Christian Schilling]

On Mon 25. Jan 2021 at 02:04, Taylor Blau <me@ttaylorr.com> wrote:
>
> On Sun, Jan 24, 2021 at 01:40:08PM +0100, Christian Schilling wrote:
> > Hi list,
> > I have been working on a custom git server where the sha values
> > required to respond to a ls-refs command can occasionally be expensive
> > to compute.
>
> Almost certainly the expensive part of ls-refs is actually traversing
> the references, not parsing the objects they point at and determining
> their hash.

My use case is very different. I’m working on a custom git server that
creates refs as well as the objects they are referring to on demand.

More specifically the references are derived from the repo url to
implement partial cloning.
So you can do:
git clone http://repository-url.git:/any/subdir/you/want.git

The server will do something similar to git-filter-branch and
deliver the result.
Due to the way my own branch filtering implementation works this is
usually fast enough for the users not to notice what is going on,
but the fact remains that the shas are only known after the filtering
is complete.
Also my server acts as a proxy with the canonical backing repository
hosted on a different machine.
So to start filtering the proxy has to first ask the upstream for the
current state of the repo and download
any new changes. I have done a lot of optimizations to make
all of this fast and in most cases it is, but for the occasional case,
for example if a completely new repo is cloned for the first time,
it leaves the users looking at a seemingly “frozen” command prompt
not knowing if they made a mistake or are encountering some bug or
network problem.

At first I also thought about suggesting a protocol extension that allows
to start transferring pack data before all refs are known to reduce the
total time needed, but the slow cases are infrequent enough so that this
probably not worth it.

> Incidentally, we had a discussion recently [1] that resulted in some
> patches that make it so that ls-refs often only has to read through part
> of the refs in your repository, not all of them.

I had my own struggles with the “to many refs” problem, but this is
unrelated :-)

>
> > It would be a great improvement of user experience if it was possible
> > to show progress to the user while this is happening.
>
> It's possible that that might help, but honestly I'd be surprised if
> there was a real use-case that needed it (especially after the patches
> that I mentioned which should make it fast enough that you don't have to
> care :-)).

Sorry for not being clearer about what I am trying to accomplish the first time,
I hope I convinced you that this is a very different, but real, use case.

Thanks,
Christian

>
> Thanks,
> Taylor
>
> [1]: https://lore.kernel.org/git/20210119144251.27924-1-jacob@gitlab.com/